Photo hosting on MyE28

[tig]

I am working on (finally!) enabling mye28.com to enable photo uploads for forum posts. This thread is for me to document how I'm doing it and to get feedback from the community.

What's Been Done So Far

As part of the infrastructure upgrade, every external link via an img tag to a photo posted prior to December 27, 2024 is now automatically be redirected to a copy of the image hosted by mye28.com.

That's 149,696 photos using 57GB of storage that will never be lost because services like Photofuckit go tits up.

The photos are stored in an Azure Blob Storage Container (the MS equivalent of an Amazon S3 Bucket for those who follow these things). That Blob Container is mounted on the mye28.com server as a file system, enabling file system access.

Prior to today, when you viewed a post, such images were pulled from a URL like this: images/ext/0b3dddde2eef0a2b81e4112f68bc2280.jpg. IOW, they were served up by the mye28.com webserver.

As of about an hour ago, they are now pulled directly from the Blob Bucket, with URLs like this: https://mye28.z13.web.core.windows.net/ ... bc2280.jpg. IOW, the Internet address of the Blob Container is https://mye28.z13.web.core.windows.net.

The great things about this scheme are:

- Storage costs are extremely low (roughly $0.020 per GB/month).
- There's no load on the mye28.com virtual machine when photos are accessed.
- It's trivial to setup inexpensive CDN support

Separately, the wiki at https://wiki.mye28.com has its own image/file store. While the Wiki is hosted on the same virtual machine as the forums, the storage is separate. I've already enabled uploads of images (and some other file types like PDF and Excel) for all users. Fles uploaded via the Wiki uploader are stored on the VM hard disk for now.

However, I've enabled the Wiki to access the same Blob Container and am already using it for large files (all the ETMs at https://wiki.mye28.com/index.php/Electrical). A future work item is to make the standard WIki uploader put images in the Blob Container not local storage.

Please DO NOT use the WIki uploader as a way of hosting photos you put into mye28 posts at this time!

Next Steps

* Figure out how to make Azure provide the right TLS certificate for https://blobs.mye28.com which will hide the ugly https://mye28.z13.web.core.windows.net (and ensure we're not tied to Microsoft's blob store forever).
* Configure the CDN
* Make the standard WIki uploader put images in the Blob Container not local storage.

Forum Photo Uploader and Storage

Now for the fun part...

Here are MY requirements. Please feel free to comment on these and suggest alternative ideas!

* Uploader must be easy to use as part of posting/editing Forum posts.
* Insertion of proper img tags should be automatic.
* Drag/Drop file upload supported.
* Multiple photoscan be uploaded at once.
* Photos that exceed the maximum allowed size will automatically be scaled with no user intervention required.
* HEIC formatted (iOS) photos get automatically converted to PNG
* Uploaded photos can NOT be deleted, except by a Beemter.
* The Beemter admin console makes it easy for Beemters to delete inappropriate photos.

Fun, but not required features I'd like to see:

* A per-user photo gallery where one can see all photos they've uploaded.

What thots do youse guise have?

[1st 5er]

I love you tig.

[vinceg101]

Charlie,
AWESOME!
Can you walk through a typical process for someone creating a post with pictures with this new system?

As to the photo host change-over, right now I have my entire M535i restoration thread hosted on Photobucket (boo), which you say shouldn't matter. However I did notice that I had quite a few photos that were not showing up (even though the link was still current back to PB). I didn't/haven't yet gone back to redo the links so I wonder if that will make a difference. (I realize I need to go back and check the whole post to see what is and is not there).
I have my own Nextcloud server that I just figured out how to host photos here and could migrate all the photos I have here to it so I could effectively drop PB but that is going to take a really long time to do. I am wondering how this is going to effect that.

Otherwise:

[tig]

Charlie,
AWESOME!
Can you walk through a typical process for someone creating a post with pictures with this new system?

Not yet.

As to the photo host change-over, right now I have my entire M535i restoration thread hosted on Photobucket (boo), which you say shouldn't matter. However I did notice that I had quite a few photos that were not showing up (even though the link was still current back to PB). I didn't/haven't yet gone back to redo the links so I wonder if that will make a difference. (I realize I need to go back and check the whole post to see what is and is not there).

If photos aren't showing up, I consider it a bug in what I've done so far. Please help me identify what WAS working that is now NOT working.

[tig]

As to the photo host change-over, right now I have my entire M535i restoration thread hosted on Photobucket (boo), which you say shouldn't matter. However I did notice that I had quite a few photos that were not showing up (even though the link was still current back to PB). I didn't/haven't yet gone back to redo the links so I wonder if that will make a difference. (I realize I need to go back and check the whole post to see what is and is not there).

If photos aren't showing up, I consider it a bug in what I've done so far. Please help me identify what WAS working that is now NOT working.

Vince, I just went through your thread

viewtopic.php?t=139465

and it appears all photos are working. Some are still coming from Photofuckit.

I think I need to run the script again to pick up more PB files.

[tig]

If photos aren't showing up, I consider it a bug in what I've done so far. Please help me identify what WAS working that is now NOT working.

Vince, I just went through your thread

viewtopic.php?t=139465

and it appears all photos are working. Some are still coming from Photofuckit.

I think I need to run the script again to pick up more PB files.

Found the problem. The script didn't deal with image URLs with query strings. These worked:

Code: Select all

[IMG]https://hosting.photobucket.com/albums/rr353/vinceg101/M535%20Front%20End%20Overhaul/.highres/FEO_Ex7.jpg[/IMG]
[IMG]https://hosting.photobucket.com/images/rr353/vinceg101/DSC_0511.JPG[/IMG]

These didnt:

Code: Select all

[IMG]https://hosting.photobucket.com/images/rr353/vinceg101/DSC_0511.JPG?width=1920&height=1080&fit=bounds[/IMG]
[IMG]https://hosting.photobucket.com/images/rr353/vinceg101/DSC_0012(1).JPG?width=1920&height=1080&fit=bounds[/IMG]

I'm now re-running the scripts. It may take a day or two...

[vinceg101]

I just went through that whole thread and didn't see any missing links either (all of those are on PB). So that one isn't a problem.
I will check my other threads (I could have sworn there were missing photos somewhere).
It may take a bit (guess I know what I'm doing while watching football tomorrow...)

[tig]

Charlie,
AWESOME!
Can you walk through a typical process for someone creating a post with pictures with this new system?

Not yet. 
 

Vince, if I can get the new post editor to work as advertised, the process will be as easy as dragging one or more photos from your Finder/Explorer into the editor, exactly as you’d do with a word processor. 

[milarsky]

My goodness.  Thank you!!!!!  I'm so excited for the best BMW board by far.

jeff

[e28Sean]

Thank you again for all the hard work you've put in.   I'm going to put out one word of caution, though; Keep in mind the liability involved in hosting images if someone uploads something illegal.  You don't want to find the site in possession of that sort of thing if the law comes knocking.

I only bring it up because I've had an issue on another site I'm involved with, wherein we had a member uploading things that were illegal in most universes.  We ended up killing all image hosting and telling our users to use something like Imgur.

[tig]

Thank you again for all the hard work you've put in. I'm going to put out one word of caution, though; Keep in mind the liability involved in hosting images if someone uploads something illegal. You don't want to find the site in possession of that sort of thing if the law comes knocking.

I only bring it up because I've had an issue on another site I'm involved with, wherein we had a member uploading things that were illegal in most universes. We ended up killing all image hosting and telling our users to use something like Imgur.

Understood.

[tig]

I just went through that whole thread and didn't see any missing links either (all of those are on PB). So that one isn't a problem.
I will check my other threads (I could have sworn there were missing photos somewhere).
It may take a bit (guess I know what I'm doing while watching football tomorrow...)

The fixed external images script just completed. Vince, your thread we were using as test above is now fixed.

All images come from the image-redirect store (http://mye28.z1...) now:

Re: 1985 Arktisblau M535i Renovation - Progression - MyE28.com

[tig]

Previously there were 150571 images in the image-redirect (external) store. Now there are 156581.

Storage used is about 81GB.

[tig]

If you think an externally linked photo should be working but isn't I really want to know about it!

For example, if you see this: 



You can do the following to help diagnose:

Right click on "Image" and select you browser's equivalent of "Inspect":


You'll see something like this:


The browser is showing you the underling HTML. In this case an <img> tag linking to an external photo at "http://memimage.cardomain.net/member_im ... 5_full.jpg".

Right click on that URL and select "Open in new Tab". In this example, if you do that you'll get an error that the image doesn't exist. This is because cardomain.net died a horrible death years ago. This image is GONE. So there's no fixing it.

If you "Open in new Tab" and the expected image shows up, please post to this thread:

- A link to the Post in question.
- The URL of the missing photo.

Thanks!

 

[tig]

This is a test. This image was uploaded using the built-in PhpBB attachments feature. Which I temporarily enabled.

mye28.png

 

[Justin_FL]

Looks like some pictures from my 518i thread are not redirecting to the MyE28 image storage. Seems to only affect the pictures on Google photos, but only a couple as most pulling from our server.

Thread: viewtopic.php?t=115493

Example image:

[tig]

Yup, a bunch of photos from googleusercontent.com didn't get downloaded by the script.

Fixing now...

I guess I need do figure out how to test this more deeply. 

[tig]

Ok googleusercontent.com should all be downloaded now, if they are good URLs.

 

[tig]

The external image cache now has 212577 photos.

[DavidE9]

Sorry if I missed it, but how do we post photos going forward ? Can they be directly uploaded to the server, or do we need to use a 3rd party ?

Thanks again for the work here.

[tig]

Sorry if I missed it, but how do we post photos going forward ? Can they be directly uploaded to the server, or do we need to use a 3rd party ?

Thanks again for the work here.

If you read the first post of this thread you'd see that you still need to use a 3rd party. Built-in hosting is a work-in-progress.

[ahab]

So in other words, the TL;DR for this is that users put their pics up wherever they want and the forum software will automatically scrape them off that server and bring them internal, updating the html along the way, thereby preserving the pics in perpetuity in the mye28 multiverse. Do I have that right?

[Justin_FL]

So in other words, the TL;DR for this is that users put their pics up wherever they want and the forum software will automatically scrape them off that server and bring them internal, updating the html along the way, thereby preserving the pics in perpetuity in the mye28 multiverse. Do I have that right?

That matches my understanding of the current setup. The built-in phpBB attachment function leaves a lot to be desired and at least with the old server it was not practical to host gigs of pictures, so just flipping the switch wasn't as simple as it sounds. I'm glad Tig has the time, skills and enthusiasm to make something better than the defaults with the power of the new setup!

[tig]

So in other words, the TL;DR for this is that users put their pics up wherever they want and the forum software will automatically scrape them off that server and bring them internal, updating the html along the way, thereby preserving the pics in perpetuity in the mye28 multiverse. Do I have that right?

That matches my understanding of the current setup. The built-in phpBB attachment function leaves a lot to be desired and at least with the old server it was not practical to host gigs of pictures, so just flipping the switch wasn't as simple as it sounds. I'm glad Tig has the time, skills and enthusiasm to make something better than the defaults with the power of the new setup!

This is not quite right. There are two scenarios:

Legacy External Photo Links
I've added an extension to the forum called image-redirect that includes scripts that, when run, go through every post on the board, identifying img tags that link to photos. For each img tag, the original external photo is downloaded to mye28's storage. Then, anytime one of those posts are accessed the etension changes the HTML served to actually get the photo from the mye28 storage. 

These scripts DO NOT RUN REGULARLY. They are intended to be run once or infrequently. I last ran them yesterday.

So any post created to an externally linked img tag after yesterday STILL comes from its original source.

This solves 2 problems:

- It ensures that externally linked photos will be safely archived and available on mye28 even if the external source (e.g. photofuckit, e28-535i.com, flickr) goes away (user deletes flickr account, etc...).

- It ensures photos linked to e28-535i.com and other ancient websites that do not not support HTTPS work, by creating a copy that IS available under HTTPS.

The MyE28 Photo Uploader
 
This does not yet exist. I am working on creating it.

PhpBB has a built in uploader called "Attachments" which sucks. We will not be using it.

Once the The MyE28 Photo Uploader is implemented the need for running the Legacy External Photo Links scripts on a regular basis will go away because most people will use the uploader for new posts. 

[ahab]

OK, thanks for the explanation (and of course the herculean effort behind it). 

[Justin_FL]

This is not quite right. There are two scenarios:

Yeah, I figured that was the extension you were using or at least something similar from what I remember after looking into it years ago. Was just trying to confirm his general theory sounded about right without nerding out, so I appreciate the additional technical details!

Anyway, I spun up a Google Cloud bucket and uploaded my old website picture contents so hopefully my post images would be grabbed (until I figure out a more permanent host, etc after the trial period). Looks like at least some pics are cached already. I also edited a bunch of old posts to point to correct links/domains where they changed from like 20 years ago, so I'm sure they will be grabbed the next time you run the script.

What I noticed and not sure if it is specific to my domain/cloud config, but the rewrites here seem to point to files without an extension. Maybe it doesn't matter?

Example image:
https://mye28.z13.web.core.windows.net/ ... 0c051eeee3

From this post:
viewtopic.php?p=863581#p863581

[tig]

What I noticed and not sure if it is specific to my domain/cloud config, but the rewrites here seem to point to files without an extension. Maybe it doesn't matter?

Example image:
https://mye28.z13.web.core.windows.net/ ... 0c051eeee3

From this post:
viewtopic.php?p=863581#p863581

That's a bug (I think).

Here's the links in that post:

Code: Select all

[img]http://www.jtdigital.com/gallery/pictures/5erFest/2003/JustinHowe/DSCN0700.jpg[/img]

The database says:



But the HTML the extension is emitting is:

Code: Select all

<img src="https://mye28.z13.web.core.windows.net/external/3ca574b92784a30a624cc00c051eeee3" class="postimage" alt="Image">

An that aint right.

If you try to go to https://mye28.z13.web.core.windows.net/ ... 0c051eeee3 directly the browser tries to DL the image. But when in the site, it renders correctly.

So it probably doesn't matter, but I want to fix it because it smells wrong to my OCD.

Hmmm...

[tig]

So it probably doesn't matter, but I want to fix it because it smells wrong to my OCD.

Hmmm...

Fixed.