I like the idea of supporting multi cloud, not cloud vendor lock-in, but I still doubt providing a tool can solve this.
We have been building a DBaaS(database as a service) for many years. Until now, we have only supported AWS and GCP, have not supported Azure. It is really hard to let your product run stably on multi clouds.
For example, not all the cloud vendors provide the same infrastructure at the same time. We wanted our database run on K8s, but several years before, only GCP provided its managed K8s - GKS, so we had to provide our DBaaS on GCP at first. We also wanted to let our customer use PrivateLink to access our service, but GCP had supported too late than AWS, so we had to provide the VPC mechanism for a long time.
Even all the cloud vendors provide the same service, there are still some differences in the details above. E.g, one customer complained us that they met a backup problem on Google Cloud Storage, they found that the speed of our backup was not the same as our product spec statement. Then we found that we only fully tested this feature on AWS S3.
So Now even our DBaaS runs well both on AWS and GCP, we still don’t have the confidence to deploy the service on Azure. We plan to devote a quarter to testing to run DBaaS on Azure in the end of this year.
IMO, I still recommend handling the interfacing with the cloud directly to let us know the clouds better. Of course there are some awesome tools helping us to do some functional encapsulation, hiding the underlying implementation, but we must ensure these tools are easy enough to maintain.
IMO, Using AI for image generation is the trend, whether we like it or not. But for me, as a software engineer, I still insist on creating pictures for my own blogs or presentations, etc.
I'm not a painter, of course I paint badly. But I find that it is not so difficult to create a picture that expresses my ideas. Mostly I try to use https://excalidraw.com/ for sketching hand-drawn, and find the free stock pictures from https://www.freepik.com/.
I think this is another joy of creation, like programming :-)
Using map or array instead of switch-case is a common optimization way, but please do a benchmark if you have no confidence whether you need do or not.
Second this. One of my favourite programming books. The advice is applicable to pretty much every domain of software engineering, and the lessons are philosophical (as the name suggests) rather than technical and specific, so they serve as a great jumping off point for many specific issues you might face in programming. It’s also very short which helps!
When we started to build the distributed database TiDB, we had the similar discussion - whether we need to provide a so strong consistence for the customers? Inspired by Google Spanner, we also wanted to let the application programmers to take easy and not struggle with the consistent problems in applications layer by themselves.
So we decided to provide the strong consistency. But unlike Google Spanner, we couldn’t use TureTime API, so we introduced a standalone timestamp service named TSO to allocate timestamp. Using TSO is simple to guarantee all transactions’ order, but this introduces another problem - if the TiDB is deployed crossing different regions, the transaction latency is high.
Another example for consistency is from our realtime analytical processing design. TiDB is a HTAP(hybrid transaction/analytical processing) database. It is normal to keep consistency in the TP part, but for the AP part, is it necessary to keep consistency? If there is an update in the table, does the customer really need to get the latest data from the AP part? Here we still choose “YES”, we want our customers to focus on their business and not worry whether the query data result is consistent of not.
Of course, keeping consistency in the database comes at a cost. Now we have been trying our best to optimize the latency and performance problems :sob:
IMO, we choose a right way. We now have supported strong consistency at first, and then we can provide a loose consistency for performance too, but on the other hand, if we only build a loose consistent database at first, it is hard to provide a strong consistency for the customers.
We build our binary first with one image as the builder image, then use `copy` to copy the binary from the builder to the final executable image like alphine.
an example Dockerfile likes:
FROM golang:1.18.1-alpine as builder
# RUN apk add, wget, etc, and build the binary
FROM alpine
# or FROM scratch
COPY --from=builder builder/binary /binary
ENTRYPOINT ["/binary"]
How do non running utilities increase attack surface? If you're able to execute inside the container couldn't you just write whatever utilities you want in?
There are many exploits that may give one the ability to execute shell commands. If there is no shell or commands to even write a file in the first place, mobility becomes limited.
once in awhile i use 'kubectl exec ' to run some commands against my running prod containers to debug something or extract an environment variable. can i still do that without a shell or anything?
That sounds sweet! But I guess DigitalOcean doesn't support it? I'm getting "error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource")."
You can’t, no. One approach is a “debug sidecar” container in the Pod that has your desired tools. Linkerd, for example, will add such a container if an annotation is present on the Pod: https://linkerd.io/2.11/tasks/using-the-debug-container/
I usually just include a shell, but to each their own.
As a chief engineer, I started to work with my colleagues to build an open source distributed, relational database from scratch 7 years ago. At that time, MySQL Sharding, or NoSQL was the popular solutions for scalability. So the first challenge for us is how to design our system architecture? Followings are something we do:
- Paper. We learned a lot from the paper, like Google Spanner, Google Percolator, Raft Algorithm, etc.
- Learn from the open source projects and Leverage the power of community. E.g, we learned storage engine from RocksDB, ClickHouse, etc. to build our own engine. We also let the community contribute ideas and codes to our product.
- Try to link people as many as you can and learn from them. E.g, when I read a paper, mostly I will try to contract the authors and have a discussion about the paper with them, at the same time, I also ask them to introduce their friends to me for further communication.
- Sharing. The more you go out and share like Meetup, Blog, Webinar, etc., the more you get.
IMO, multi cloud is the future, just as eggs can’t be in one basket. I’ve been building a distributed database for more than 7 years, and find that many of our customers had been using different databases in different cloud vendors before they migrate to our product. E.g, one customer used Azure MySQL, Cloud SQL for MySQL, Aurora for MySQL.
Another thing I find vendor lock-in is not nice, for example, one customer, which is also a big customer of one cloud vendor, has suffered a lot from the cloud vendor’s own database. The customer finds the database it is not stable when its data size grows, has to spent a long and tough time to upgrade to the new version regularly, couldn’t get a good-enough service it expected before.
IMO, not only for our database, but also for other applications on the cloud, supporting multi cloud is mostly a MUST. And we also need to build an easy-to-use product, provide a good-enough service for our customers.
I've been building a distributed database product for more than 7 years, managing a team of nearly 300 R&D staff and serving hundreds of customers.
Customer Support is hard for us. Not only the number of OnCalls, but also the complexity of the product causing that we can't solve the problem easily and fast. We are under a lot of pressure.
To solve this, I have tried following ways:
- Build an independent team (may name Escalation Engineering or L3 in other companies, here let's name L3) to focus on the customer problem solving.
- Establish the OnCall duty mechanism, some engineers (maybe 10 one time) must join the L3 for two weeks only focusing on OnCall. Why it is so important, because I believe the quick way to learning our product is from OnCall.
- Get to know our customers better. We are a database product, if we can know our customers' business workloads, we can give them more advices to let the business application run more better in our product.
- Build many Diagnostic tool to solve the OnCall easily.
Finally, I would like to say that although customer support is sometimes painful, it allows us to better understand our customers and know the strengths and weaknesses of our products.
Very sad to hear this news, but not so unexpected.
As a distributed database engineer, we have been working with some universities and exploring the optimization of database on Intel Optane for a long time, and even have published some papers.
But everything was only in the experimental stage, it was difficult for us to use the research successfully in production, because nearly all of of our customers hadn't bought Intel Optane. They thought Optane was too expensive and didn't find suitable scenarios. As far as I know, the only successful story was from one of our customer, he used Optane for Redis which had an acceptable latency and better cost-effective.
Things go worse when the cloud is raising. We find that we can't use Optane on the cloud, so our later priority is to explore the optimization on the cloud disk like EBS GP3, etc.
We have been building a DBaaS(database as a service) for many years. Until now, we have only supported AWS and GCP, have not supported Azure. It is really hard to let your product run stably on multi clouds.
For example, not all the cloud vendors provide the same infrastructure at the same time. We wanted our database run on K8s, but several years before, only GCP provided its managed K8s - GKS, so we had to provide our DBaaS on GCP at first. We also wanted to let our customer use PrivateLink to access our service, but GCP had supported too late than AWS, so we had to provide the VPC mechanism for a long time.
Even all the cloud vendors provide the same service, there are still some differences in the details above. E.g, one customer complained us that they met a backup problem on Google Cloud Storage, they found that the speed of our backup was not the same as our product spec statement. Then we found that we only fully tested this feature on AWS S3.
So Now even our DBaaS runs well both on AWS and GCP, we still don’t have the confidence to deploy the service on Azure. We plan to devote a quarter to testing to run DBaaS on Azure in the end of this year.
IMO, I still recommend handling the interfacing with the cloud directly to let us know the clouds better. Of course there are some awesome tools helping us to do some functional encapsulation, hiding the underlying implementation, but we must ensure these tools are easy enough to maintain.