Hello, fellow lemmings!
I have three updates regarding lemm.ee:
Image uploads
As some of you have already noticed, image uploads are currently not working on lemm.ee at all. The short explanation is that I have been trying to figure out the best way to enable image uploads without incurring big additional costs in terms of storage, bandwidth, and moderation.
I have come to the conclusion that initially, the most sustainable path will be to enable image uploads only for small images (<100kb) - this should be enough to allow folks to upload avatars for themselves and their communities, but it for sure won’t allow uploading many bigger images or gifs for your posts.
For bigger images, I would ask everybody to use an external image hosting service for now, as was usually done on Reddit before they added image uploads.
I am in the process of creating a patch to the image upload code that will allow all Lemmy instance admins to configure a smaller limit on the uploaded image sizes. Once that patch is finished and accepted, I will be deploying it on lemm.ee - at that point, you will be able to finally upload avatars. Sorry for the wait, hopefully it won’t be much longer!
“No pornography” rule
There is a new rule landing in your sidebar - “No pornography”. I was initially against adding any too-restrictive rules, but after many discussions with other instance admins, I have realized that there is a big amount of additional risk to a community when allowing pornography.
As of writing this post, there are no pornography-related communities on lemm.ee, so I’m hopeful that this rule won’t be too controversial. If you were planning to create such a community on lemm.ee, I’m very sorry to ruin your plans and I hope you can find another instance that is more suitable for your community.
Growth of lemm.ee
The rate of new users joining lemm.ee has certainly been much faster than I expected. For now, the growth has been sustainable, and we can still keep growing at this same rate for a while longer without incurring additional costs.
However, if this growth continues for an extended period of time, then additional costs will become unavoidable. At that point, we will have the option of either closing sign-ups to new users, or upgrading our infrastructure to allow continued growth. The latter can only be achieved through some kind of community funding, so I will be asking the community for input if that time ever comes.
I want to be clear about one thing: if we end up closing new sign-ups, then this will not close your communities to new members. As always, Lemmy users from any other instance will be able to subscribe to your communities on lemm.ee, they will be able to participate as normal and even become moderators. The only thing that will change if we close sign-ups is that new users will not be able to choose lemm.ee as their home instance.
That’s all for now, let me know in the comments if you have any feedback or thoughts about any of this!
You must log in or register to comment.
+1 to the no porn rule. I seriously considered hosting my own instance, but balked at the liability especially in regards to porn. I’m no prude, but having adult material seems like a serious headache.
First of all, thanks for setting up this instance!
I’d suggest to open some form of paid subscription right now, not waiting for the moment when we’ll need to make hard choices. And don’t bother with asking for single-time donations - it’s not sustainable with so small number of users.
@sunaurus single handedly help me beat porn addiction.
Thanks for looking out brother!
To me - no pornography is even beneficial
Hey, thanks for creating the instance and keeping it ship shape!
Just a couple questions about timelines
When do you expect the patch for image posting might be done? - I’m not knowledgeable at all about any of this stuff, does the patch mean we’ll be able to post images and they’ll show up organically/embedded as part of the post? (As opposed to seeing a link to Imgur where you can’t see the actual image)
Re growth of lemm.ee - How long would you expect the ‘extended period of time’ to be if growing rate stays constant? It’s exciting to think we could be growing so much/fast!
Edit formatting
When do you expect the patch for image posting might be done?
I’m hoping during this week, but can’t guarantee anything. I’ll keep you guys posted.
does the patch mean we’ll be able to post images and they’ll show up organically/embedded as part of the post?
Yes, as long as the images are <100kb 😀
How long would you expect the ‘extended period of time’ to be if growing rate stays constant?
It’s difficult to predict, as we’ve only been live for a few days, and there are many moving parts here. We have several pieces of our infrastructure which are being affected at different rates by the growth, and only time will tell which one gets close to the limit first.
For example, I am paying for:
- Storage for uploaded media
- Storage for the database (all posts, user data, comments, content from other instances, etc)
- Bandwidth
- E-mails
- Just plain server resources like CPU/memory for multiple servers
I’m keeping a close watch on resource usage for now, I’ll definitely keep updating everybody as time goes by.
Glad to hear you’re keeping up with all this new traffic!
Are you willing to talk about the costs you’re incurring to run this? Or some details about your deployment? I’m considering spinning up an instance and curious to get a sense of what the overhead is like. I’m guessing you’re handing this kind of traffic on…like a single EC2 instance+RDS?
Also, still trying to understand ActivePub – is it mirroring all content from all other instances locally?
details about your deployment
It’s not on AWS, but another cloud provider. Currently it looks like this:
- Cloudflare for DNS (and all the Cloudflare protections)
- Load balancer
- 2 statless servers for the Lemmy software
- 1 stateful server for image upload software pict-rs (which is not yet usable by users)
- s3-type object storage for uploaded media
- RDS-type managed database
- Terraform cloud for infrastructure orchestration
- Grafana cloud for monitoring
- Some mail service for sending e-mails - not settled on a provider yet, it might be Postmark, but their support has been less than helpful so far, so it might end up being something else as well
ActivePub – is it mirroring all content from all other instances locally?
Yes! Everything is cached locally, so even if other instances go down, their content will still be visible here (and vice versa).
Thank you so much for the details!!!
Re: ActivePub (If I’m not imposing), what does your current storage volume/growth look like? My long-term concern was that, at some point, smaller instances with few users wouldn’t be able to keep up with the cost of data growth or sync overhead. But it does at least look like media is requested from the instance it was created on no matter what instance you’re browsing.
The database is at around 400mb currently, and most of that is in a big table that is constantly getting cleaned up. Posts and comments are just text, so I don’t expect it to grow out of hand too quickly (hopefully).
I’m much more worried about image uploads. For example, the default image size limit that Lemmy ships with is 40mb. You can imagine how quickly a few abusive users could fill up an instance with 40mb images (not to mention use up a lot of bandwidth), but even without abuse, if a bigger instance has hunderds of users constantly posting images, the storage costs will add up very quickly.
Indeed. I’m just sampling some from r/pics, even the largest/highest-res photos I saw clock in at ~1MB or just over (jpg, webp). I’m going to dive into Lemmy’s source and see what they’re doing… I imagine the 40mb is an upload limit on the pre-compressed filesize (still enormous). It would be ideal if they shipped with sliders for “compression aggression” or let you pass flags into ImageMagick or whatever it’s using. Even 1MB/image would get expensive fast unless you could prune old images or move into a cheaper cold storage product.
I’ve definitely seen some >10MB files in places like /r/EarthPorn and various wallpaper threads. That’d definitely get impactful fast. Cloudflare will help, especially if you can set nice long TTLs for caching. I’m glad S3 type storage is something that works with Lemmy, I was actually wondering this myself. You could set that to Infrequent or “Cold” access polices after a day or so to throw it into a cheaper bucket for long term storage being that Cloudflare will take the brunt of the hits.
Side note: It warms my DevOps heart to hear this setup. Definitely sounds scalable and easily duplicated or redeployed if need be. Major props and I do thank you for this setup!
Question: I haven’t looked into it, but why does the upload server have to be stateful? Perhaps it could be a lambda or similar?
However, if this growth continues for an extended period of time
What do you expect the bottleneck to be as the platform/server grows? Is it user accounts taking up space, different people subscribing to different federated servers or the cpu/bandwidth for serving users? Or something else?
Allow me to copy my response to a very similar question from another thread:
It’s difficult to predict, as we’ve only been live for a few days, and there are many moving parts here. We have several pieces of our infrastructure which are being affected at different rates by the growth, and only time will tell which one gets close to the limit first.
For example, I am paying for:
- Storage for uploaded media
- Storage for the database (all posts, user data, comments, content from other instances, etc)
- Bandwidth
- E-mails
- Just plain server resources like CPU/memory for multiple servers
I’m keeping a close watch on resource usage for now, I’ll definitely keep updating everybody as time goes by.
