Amazon’s S3 (Simple Storage Service) isn’t new, but its certainly gaining traction. Its a wonderful product for people who have a lot of content on their site (images, video, downloads, pdfs) but not a lot of money. Data storage costs $0.15 per GB-Month (prorated), and $0.20 per GB. No minimums, rounded up to the nearest cent.
There are a lot of great providers out there (I use Laughing Squid and highly recommend it) but even LS’s ‘largest’ package is too small for ladyada.net… What to do? Easy: Host all that bulky content at S3, then use mod_rewrite to reroute it over to S3. (You could also do it with php, asp or similar for higher ’security’ but mod_rewrite is lighter and good enough for me)
For example, this image has the url reference “http://www.ladyada.net/images/mintyboost/assemblyv12/inductorusbplace_t.jpg” but if you access that url in your browser, it is automatically rewritten by apache to http://s3.amazonaws.com/ladyadanet_mintyboost/assemblyv12/inductorusbplace_t.jpg
(same with my research pdf, a big pdf that easily accounted for 500M a day of traffic at its peak! http://www.ladyada.net/media/common/thesis.pdf -> http://s3.amazonaws.com/ladyadanet_common/thesis.pdf , S3 doesn’t care what the data is or how its encoded)
Of course mod_rewrite is not necessary, you can always just directly reference s3.amazonaws.com but that makes it harder to move the content around if you decide to eventually go with another service (or if s3 goes away one day!)
OK so, what’s the point and what does this have to do with electronics, eh? Well one of the killer apps of open source and public domain electronics is documentation. That means media. And media storage, backup and transfer is extremely expensive for the everyday person. It becomes increasingly difficult to host a project when one digg-storm or slashdotting makes that ‘free’ webpage account go down.
Edit: I use the Firefox S3 plugin to upload and set the access control on my files.
Are you using S3 or something similar for your projects, kits or documentation? Leave a comment or email! Its always interesting to see what other people are doing in this space.
Adafruit publishes a wide range of writing and video content, including interviews and reporting on the maker market and the wider technology world. Our standards page is intended as a guide to best practices that Adafruit uses, as well as an outline of the ethical standards Adafruit aspires to. While Adafruit is not an independent journalistic institution, Adafruit strives to be a fair, informative, and positive voice within the community – check it out here: adafruit.com/editorialstandards
Stop breadboarding and soldering – start making immediately! Adafruit’s Circuit Playground is jam-packed with LEDs, sensors, buttons, alligator clip pads and more. Build projects with Circuit Playground in a few minutes with the drag-and-drop MakeCode programming site, learn computer science using the CS Discoveries class on code.org, jump into CircuitPython to learn Python and hardware together, TinyGO, or even use the Arduino IDE. Circuit Playground Express is the newest and best Circuit Playground board, with support for CircuitPython, MakeCode, and Arduino. It has a powerful processor, 10 NeoPixels, mini speaker, InfraRed receive and transmit, two buttons, a switch, 14 alligator clip pads, and lots of sensors: capacitive touch, IR proximity, temperature, light, motion and sound. A whole wide world of electronics and coding is waiting for you, and it fits in the palm of your hand.
Have an amazing project to share? The Electronics Show and Tell is every Wednesday at 7:30pm ET! To join, head over to YouTube and check out the show’s live chat and our Discord!
Python for Microcontrollers – Adafruit Daily — Python on Microcontrollers Newsletter: New Python Releases, an ESP32+MicroPython IDE and Much More! #CircuitPython #Python #micropython @ThePSF @Raspberry_Pi
EYE on NPI – Adafruit Daily — EYE on NPI Maxim’s Himalaya uSLIC Step-Down Power Module #EyeOnNPI @maximintegrated @digikey
I would also recommend Dreamhost for hosting. Their basic package ($9.95/mo) gives you 1.654 TB (yes, Terabytes) of bandwidth a month which increases by 16GB every week. Disk storage is 165.4 GB and increases by 1GB/week.
Also, why would you not zip up that PDF? Unless it’s just images jammed into PDF it will probably compress down to a minuscule shadow of it’s former self. Sure it won’t open in a browser if it’s zipped, but whatever.
Finally, I think you’re a victim of your own success. Have you considered simply being less popular?
Well the PDF was generated by InDesign, could probably be compressed some. But even if I fixed that, there’s hundreds of images (and multiply that by getting linked from a ‘popular’ blog) Its more a question of scale: for every new project I churn out theres a bunch of documentation added to an existing pool of content. And nobody would like it if i skimped on my documentation or took down older stuff!
As for dreamhost…well, they dont get such great reviews: http://www.webhostingjury.com/reviews/DreamHost . Sounds like they jam a lot of servers onto one machine: great for everyday users, but not great for people trying to keep the website up and running with so much stuff 🙂 (One of the nice things about LS is that they co-lo at Rackspace which is ridiculously reliable.)
Anyways with S3 you dont have to worry about how fast DH can push out data or whether its stored concurrently – its ultrafast already and spread all over Amazon’s huge network!
Thanks for the comment – Im really interested in this ‘problem’, perhaps enough to think about how something like ‘sourceforge’ for projects could be super useful (instructables.com approaches this right now)
I’m using ADSL with 4Mb output (i think 512k input) with static IP with my own “in the house” server.
The cheapest solution (at least for now) regarding the cost.
i am using http://www.pixilive.com.
Pixilive is a free image hosting service. Upload your pictures, images or videos and share them easily with your friends, family, or on Forums, messenger mail, Blogs, Avatars, etc.. This service is completely FREE!!
I could not agree more with your dislike of Dreamhost. They are the definition of over selling, and as you mentioned they have a nice habit of jamming far too many virtual servers onto a single physical server. Neither of these traits are much to my liking when it comes to a webhost. I don’t exactly have a large blog, but back a few months ago I was pushing past my bandwidth every single month. Although I could have moved all of my images over to S3, I decided at the time to move them over to Flickr to reduce the bandwidth. My planned paid off, but I am still looking at possibly switching to S3 instead.
for a less than five bucks a month I host five sites, and get charged only for bandwith and storage used. 1 cent a month per MB storage, and $1 a month per GB. In comparison with S3, that might seem expensive, but… the speed at NFS is faster and you can set up as many sites as you want with PHP, SSH and MYSQL, and the support is lightning fast and amazing… They also sell bandwidth in advance in “buckets”… for when you get slashdotted: $40 buys 72GB for use in the coming week, or if you just need 20GB bandwidth for two days, that’ll be $10. The main aspect fo their business is that they are not out to gouge you like mosts hosts are, but rather to give you exactly what you need. this is reflected in little things like automated balance (i.e. traffic) warnings, free awstats, and pretty much total control over the innards of your hosting.
Probably not everyone’s cup of tea, but it is kind of addictive once you start using it. Definitely worth checking out.
For DNS try zoneedit.com. It hosts upto five domains for free. Lets you create A, CNAME, MX etc records.
Easy to use web interface.
But the mod-rewrite approach looks more powerful to me.
I will be looking around for more info on that.
I would also recommend Dreamhost for hosting. Their basic package ($9.95/mo) gives you 1.654 TB (yes, Terabytes) of bandwidth a month which increases by 16GB every week. Disk storage is 165.4 GB and increases by 1GB/week.
Also, why would you not zip up that PDF? Unless it’s just images jammed into PDF it will probably compress down to a minuscule shadow of it’s former self. Sure it won’t open in a browser if it’s zipped, but whatever.
Finally, I think you’re a victim of your own success. Have you considered simply being less popular?
Well the PDF was generated by InDesign, could probably be compressed some. But even if I fixed that, there’s hundreds of images (and multiply that by getting linked from a ‘popular’ blog) Its more a question of scale: for every new project I churn out theres a bunch of documentation added to an existing pool of content. And nobody would like it if i skimped on my documentation or took down older stuff!
As for dreamhost…well, they dont get such great reviews: http://www.webhostingjury.com/reviews/DreamHost . Sounds like they jam a lot of servers onto one machine: great for everyday users, but not great for people trying to keep the website up and running with so much stuff 🙂 (One of the nice things about LS is that they co-lo at Rackspace which is ridiculously reliable.)
Anyways with S3 you dont have to worry about how fast DH can push out data or whether its stored concurrently – its ultrafast already and spread all over Amazon’s huge network!
Thanks for the comment – Im really interested in this ‘problem’, perhaps enough to think about how something like ‘sourceforge’ for projects could be super useful (instructables.com approaches this right now)
I’m using ADSL with 4Mb output (i think 512k input) with static IP with my own “in the house” server.
The cheapest solution (at least for now) regarding the cost.
i am using http://www.pixilive.com.
Pixilive is a free image hosting service. Upload your pictures, images or videos and share them easily with your friends, family, or on Forums, messenger mail, Blogs, Avatars, etc.. This service is completely FREE!!
@ladyada
I could not agree more with your dislike of Dreamhost. They are the definition of over selling, and as you mentioned they have a nice habit of jamming far too many virtual servers onto a single physical server. Neither of these traits are much to my liking when it comes to a webhost. I don’t exactly have a large blog, but back a few months ago I was pushing past my bandwidth every single month. Although I could have moved all of my images over to S3, I decided at the time to move them over to Flickr to reduce the bandwidth. My planned paid off, but I am still looking at possibly switching to S3 instead.
I’m using NearlyfreeSpeech… http://www.nearlyfreespeech.net .
for a less than five bucks a month I host five sites, and get charged only for bandwith and storage used. 1 cent a month per MB storage, and $1 a month per GB. In comparison with S3, that might seem expensive, but… the speed at NFS is faster and you can set up as many sites as you want with PHP, SSH and MYSQL, and the support is lightning fast and amazing… They also sell bandwidth in advance in “buckets”… for when you get slashdotted: $40 buys 72GB for use in the coming week, or if you just need 20GB bandwidth for two days, that’ll be $10. The main aspect fo their business is that they are not out to gouge you like mosts hosts are, but rather to give you exactly what you need. this is reflected in little things like automated balance (i.e. traffic) warnings, free awstats, and pretty much total control over the innards of your hosting.
Probably not everyone’s cup of tea, but it is kind of addictive once you start using it. Definitely worth checking out.
Why not use a CNAME record and avoid the rewriting entirely?
flexibility, and also i dont have DNS access at my hosting provider
For DNS try zoneedit.com. It hosts upto five domains for free. Lets you create A, CNAME, MX etc records.
Easy to use web interface.
But the mod-rewrite approach looks more powerful to me.
I will be looking around for more info on that.