Hacker Newsnew | past | comments | ask | show | jobs | submit | tbarbugli's commentslogin

At getstream.io we ended up running Github Actions on Hetzner. The end-result is 4x faster builds for 3x less $$$.

Running workers ourselves was the last resort, we tried everything else but it was impossible to get fast (and consistent) build times otherwise.

In a way we are now going to get charged for Github's poor execution on Actions.


How are you guys running it? Is it via RunsOn, Ubicloud? We just moved ours to Blacksmith since I still don't want to manage the worker ourselves yet. But with this change, we might be looking into cheaper, better alternatives if there's any.


> anyone can see what URLs you are visiting

this is not correct with HTTPS (query params are not part of the plain text)


ofc it depends where you would hire, for me (NL) its above 100x more efficient


I agree, only reason I read the project readme was to see the drop-in explainer.

Very misleading title


cc klaussilveira


Well, it says right there in the GitHub About section: "Drop-in replacement for E in the ELK stack".


searching for running shoes returns a mix of brands and shoe models


Plenty of engineers on the market can work professionally in C. Only a small amount of people can write Cobol (or is willing to given that is almost useless). That alone is a good reason to consider Cobol a legacy language and throw away a codebase written in Cobol.


Consider legacy: yes.

Throw away the codebase: no, until you have a well-working, battle-tested replacement. I mean, yes, it's possible to do, and you can even run e.g. Java under z/OS on the same in-house mainframe hardware because you can't trust the public cloud. But you still have to do the massive work of reverse-engineering the ancient, scantly documented Cobol codebase, write a modern replacement, cover a ton of corner cases, run in in prod as a shadow that does all the same work, and comparing the results with the load-bearing Cobol codebase, and switch over very carefully.

Depending on the size of your codebase, the complexity of your processes, the strength of your need for change, and the quality of your engineering org, the above may be a very costly process. Few managers are comfortable to approve such a large cost without a very clear return on this investment. Hence we'll see Cobol running for a few decades more.


> At Compose, every WebSocket message starts with a fixed 2-byte type prefix for categorizing messages.

some of the complexity is self-inflected by ignoring KISS principle


How would you make it simpler?


In my experience, only Swift has a generator that produces good-quality code. Ironically, it’s developed by Apple.


The list of supported commands looks quite impressive https://kvrocks.apache.org/docs/supported-commands#script-co...


Using DynamoDB in 2025 is such a weird proposition. Horrible dev experience, no decent clients/libs, complex pricing, weird scaling in/out mechanism, slow, it only works well for well defined use-cases.


2 times I have used DynamoDB and been extremely happy;

- In a SAAS API service we used dynamodb to look up API keys and track their daily usage data. It is fast enough to look up k/v pairs (api key => key info). And also aggregate small sets (We'd sum up call counts for current month and check if the API key had enough credits). This meant that the API itself did not need our RDBMS to function. We also had a postgresql instance for all relational data, subscriptions, user info etc. Had a trigger that would push any api key / subscription change to DynamoDB. In case of RDS issues, things kept chugging along.

- Working on a large buzzfeed like social media / news site in my country. We needed to store a lot of counters (reactions to articles, poll answers etc). All went into dynamodb and looked up from there. No hits on actual rdbms. There were a lot of traffic and dynamo made scaling things / keeping rds from melting easy for this kind of non critical data.

I'd not build an entire thing on DynamoDB but for specific use cases, I just loved it.


> We also had a postgresql instance for all relational data, subscriptions, user info etc. Had a trigger that would push any api key / subscription change to DynamoDB.

Wouldn't doing it right there in postgres limit your footprint?


We did not want postgres to be a central failure point of this API.

Needed a pretty high uptime guarantee so we decided that as long as AWS region is up and running, the API would also be available by using only completely managed aws services like dynamodb, lambda etc. Also had a bunch of beefy servers around other providers (hetzner, online.net etc) handling the actual work. They did not have any other dependencies either.


Redis?


What would we gain from Redis in these use cases?

We used it extensively on the second project I mentioned and a couple of other projects for caching / rate limiting and distributed locking needs. Never enabled the persistence layer (which I believe is pretty durable). So we only treated as an ephemeral data store, lowering the architectural complexity of things significantly. Otherwise you need to think about backups, testing backups, clustering in case of scaling needs, I have no idea how persistence works with clustering... DynamoDB is fully managed and solid.


There are redis offerings that are fully managed as well. You have both options.


is it as easy to make that data durable?


Way too many teams choose Dynamodb too soon. Scalability, 0 management, coolness whatever. They don't realise until its too late that their application data needs are changing with feature requests and that with Ddb it implies doing 3D-chess each time to ensure the denormalised data is re-arranged the right way, rather than just using PostgreSQL with JSONB and adding an index, until/if it gets to FAANG scale, a bridge you can safely cross much later on.


Very often I find myself wanting to store item(s) using a key.

My items are not relations, and I don't see the point in transforming them to and from relational form. And if I did, each row would have like 5 columns set to NULL, in addition to a catch-all string 'data' column where I put the actual stuff I really need. Which is how you slow down an SQL database. So RDBMS is no good for me, and I'm no good for RDBMS.

RDBMS offers strong single-node consistency guarantees (which people leave off by default by using an isolation level of 'almost'!). But even without microservices, there are too many nodes: the DB, the backend, external partner integrations, the frontend, the customer's brain. You can't do if-this-then-that from the frontend, since 'this' will no longer be true when 'that' happens. So even if I happen to have a fully-ACID DB, I still lean into events & eventual consistency to manage state across the various nodes.

Given that I'm using more data than a naive CRUD/SQL app would (by storing events for state replication) and my data is stringy enough to kill my (and others') performance. So what's the solution? Make my read-writes completely independent from other read-writes - no joins, no foreign keys, etc.

The thing that would put me off using DynamoDB is the same reason I wouldn't use any other tech - can I download it? For this reason I'd probably reach for Cassandra first. That said I haven't looked at the landscape in a while and there might be much better tools.

But it also wouldn't matter what I want to use instead of DynamoDB, because the DevOps team of wherever I work will just choose whatever's native&managed by their chosen cloud provider.


> The thing that would put me off using DynamoDB is the same reason I wouldn't use any other tech - can I download it?

Amazon provides a downloadable version for development. I don't know how close it is to the real thing, but it makes it easier to do local dev.

Localstack also supports it in their paid version


The downloadable version is nowhere near ready for production. It's performance is also excruciatingly slow.


It doesn't really make any sense to use it locally - the whole point is that it's managed. If you just want a clustered key value store you could use Cassandra, Garnet, etc.


I think people mostly use it for unit testing functionality since it’s generally a faster dev loop compared to running integration tests.


It's not supposed to be used for production, it's supposed to be used for development


> I still lean into events & eventual consistency to manage state across the various nodes.

You can get really far with a RDMS before event sourcing etc is needed, the benefit being both your dev and user experience are going to be much simpler and easier.

If you already know your problem domain and scaling concerns up front sure. But starting with a scalable pattern like this is a premature optimization otherwise and will just slow you down.


> You can get really far with a RDMS before event sourcing etc is needed

You can manage up to 0 partners easily. Once you go above that threshold, you're into "2-Generals" territory. At that point you're either inconsistent, eventually-consistent, or you're just bypassing your own database and using theirs directly.

> dev and user experience are going to be much simpler and easier.

I have objects, not relations. I'm not going to do the work of un-nesting a fat json transaction to store it in a single relation (or worse, normalise it into rows across multiple tables).


Yeah this is a much better initial dev experience but you still have a schema, even if you ignore it. When your objects are inconsistently shaped something has to fix them. That something is going to take the shape of custom code that would make even the Perl-iest DBA blush.


So we've shifted from:

  SQL now (for dev experience) && no-SQL later (for scaling)
to:

  no-SQL initially (for *much better* dev experience) && no-SQL later (for scaling)
I can get behind that.

> When your objects are inconsistently shaped something has to fix them

They have one schema (the class file) instead of two (the class file and the SQL migrations).


> They have one schema (the class file) instead of two (the class file and the SQL migrations).

But what happens when that schema defining class file needs to change? You put all your migration code there? How is that different from SQL migrations?


Some RDBMS only offer single node consistency but others can scale write masters horizontally (e.g. Oracle).


> Horrible dev experience, no decent clients/libs, complex pricing, weird scaling in/out mechanism, slow, it only works well for well defined use-cases.

Most of these arguments probably don't outweigh the benefits. If you're in need of a managed, highly-consistent, highly-scalable, distributed database, and you're already an AWS customer, what would you use instead?


Aurora Serverless Postgres e.g.


Completely different use cases....


Postgres running in a computer.


> it only works well for well defined use-cases.

Sounds to me DynamoDB works well for well defined use-cases. That to me is a plus!


Exactly this.

DynamoDB is a pain in the ass if you want to do too many relational or arbitrary queries. It's not for data exploration.

It is my favourite database though (next to S3)! For cases where my queries are pretty much known upfront, and I want predictable great performance. As Marc Brooker wrote in [1], "DynamoDB’s Best Feature: Predictability".

I consistently get single digit millisecond GETs, 10-15ms PUTs, and a few more milliseconds for TransactWriteItems.

Are you able to complex joins? No. Are you able to do queries based on different hash/sort keys easily? Not without adding GSIs or a new table. The issue in the past few years was the whole craze around "single-table design". Folks took it literally as having to shove all their data in a single table, instead of understanding the reason and the cases that worked well. And with ongoing improvements of DynamoDB those cases were getting fewer and fewer over time.

But, that's what tradeoffs are about. With on-demand tables, one-shot transactions, actually serverless storage/scaling, and predictable performance you get very, very far.

1. https://brooker.co.za/blog/2022/01/19/predictability.html


> DynamoDB is a pain in the ass if you want to do too many relational or arbitrary queries

You are using it wrong. And no, it's not irony.


I’ve used dynamodb before but most people have no idea what they’re talking about re: it and start blaming the tool.

Like when we implemented it me and my colleague spent a couple days understanding single table design and how to handle the access patterns we wanted to support.

Trying to explain this to paper smart but lazy colleagues who then skipped understanding and went straight to implement something wrong and blamed the tool really opened my eyes.

For us dynamo made sense. We were tracking global quantities and things like that. Didn’t need to be real time but did need to be present across regions fast.


> It only works well for well defined use-cases.

This strongly hints at a misunderstanding of the purpose of a NoSQL system.


Scaling in and out is too slow to be usable. DynamoDB is great though, for many of my use cases it's a very good fit.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: