This is very different at the moment, not only you could have patch updates for the modified properties but you could also perform rollback in case something is rejected by the server and so on and on.
I would build a rest service that provides the code to do this, and I would test it extensively before releasing it as the mechanism to effect the change.
In addition I would build a validation service that checks and tests the update data (the list of changes) before they are submitted to the change service.
I would not permit any adhoc changes to the prod database. The rest service should be the only way for a mutation to be done short of a complete new release.
I'll stipulate that you might want to be very conservative about doing this with a production system.
No kidding. I knew at least one guy who hosed a product launch by doing that. You even have to restart your development instance from time to time to make sure that your application state remains consistent. I've been scared to try, but I suspect that you could live-patch a server though if you were systematic about it and tested the patch ahead of time.
Upgrading software and Postgres databases is a difficult problem in general.
The text itself can be changed but it takes at least 15 minutes to mutate to a deployment. We still cannot generically mutate running code to other running code. Ksplice and other live kernel patching and Chrome's binary patching should be generalised and productised.
Patching methods at runtime is another thing that is possible (ruby, python and Erlang) but I am not aware of a general framework for deploying mutations to servers at runtime.
You could design a contract that could be bug-fixed and upgraded provided some % of stakeholders consent to the upgrade. These platforms are still in their infancy so upgradability or even just thorough testing and auditing is rare.
You’re conflating unattended-upgrades (server mutability, hard to roll back) with automated patching in general. Do automated patching but also run the changes though your CI so you can catch breaking changes and roll them out in a way that’s easy to debug (you can diff images) and revert.
I bet when you update your software dependencies you run those changes through your tests but your OS is a giant pile of code that usually gets updated differently and independently because mostly historical reasons.
Then updates to the customer's configuration requires modifying code, compiling it, and shipping the result. It's easy to put an interface on configuration file modification and in many cases, updates can be applied without needing to restart a running system.
you can discuss the changes to be pushed, you can submit pull requests yourself and if you are dissatisfied about what gets accepted, you can also run your own version of the codebase with different implementations (lacking updates) as long as they aren't incompatible with the other nodes. (ie a subset of features instead of different features)
if you need me to elaborate on the ramifications/benefits of that let me know.
That’s pretty sweet. Makes me think about software upgrade scenarios. A typical upgrade process for long-running services I’ve seen is usually some variant of “flush everything out to disk; shutdown process; run some hand-crafted upgrade code for the on-disk datastructures; start the new process.”
On the other hand, the development for this stuff usually consists of a sequence of small patches. You could generate a lot of that on-disk upgrade code by applying something like this against all those patches. Maybe you could take it a step further and never actually tear down the process (by doing that memory mapping stuff the article touches on).
I wonder how far you can take that concept. People develop software today with an edit -> build -> deploy cycle. With sufficient tools, could one do development solely by describing how to patch the running process? Your repository consists of a sequence of such patches: CI applies the patches against a running process (and then runs all your tests). If that passes, you can deploy the patch everywhere using an identical procedure.
Fixes the issue of customers running outdated software because they can’t handle the one minute of downtime associated with an upgrade to a critical service. You don’t need to keep years-old upgrade code in the codebase (and keeping it well-tested) because one of those customers might want to upgrade someday. On the other hand, pulling bad patches caught late in the cycle becomes more difficult. And also the problem of downtime already has workarounds today, usually by making use of redundancy (which remains valuable even outside of upgrades).
I dunno, it’s fun to think about where and how you could apply this. Hot-reloading structs is a totally novel idea for me :-)
Optimistic updating needs a serious amount of code and robust error handling on the client and is therefore non-trivial.
This approach seems aimed to reduce the amount of work while being relatively fast.
Patching is relatively easy to solve with immutable infrastructure. Build a new fully patched AMI every night and when you release you pick (version of code, version of AMI) to deploy.
I don't really see a benefit in updating existing instances in this manner. Launching replacement instances with the new code is much easier for us, and it also provides a super fast means of rollback.
unless the update requires version updates on multiple services which means the versioning and rollback has the same effect as a more monolithic codebase. but with the added complexity of not knowing how it all impacts each other.
In my experience updates to smaller services are often trivial, or for updates that are actually impactful it would be way easier to coordinate in a monolithic codebase.
Versioning, feature toggle and rollback - automated and implemented at different level based on the system. It could be an env configuration, or db field or down migration scripts or deploying last working version.
It's fairly common to deploy a proxy contract, which allows for a sort of updating. Basically you deploy a new contract and have the proxy point to that new contract instead of the old one. There are definitely some limitations and it's not fully standardized yet. There's a tentative standard here https://github.com/ethereum/EIPs/issues/1538 and OpenZeppelin also has their own implementation.
Although neither of these things will actually help if your contract gets exploited before you can push a fix.
When I was there (several years ago) it was rare to do that across many projects. You don't want to have to roll back everyone due to a problem that affects one project. Also, there's a risk that changes that are happening in the meantime might mean the patch doesn't apply.
Transactions (something like fsync) and code upgrades (something like database migrations or hot-reload in a debugger) seem like the tricky parts and I don't see a discussion of how they'll handle it.
Let's assume you get a stream of security patches from somewhere. How do you apply them without a reboot? What if a data structure needs to be migrated?
During early development, it's easy to just throw away all your data and start over after a redesign, but as with production databases or filesystem implementations, once you're storing real user data, you're not allowed to do that anymore. It's helpful architecturally to have a distinction between on-disk structures (which you're careful about either not changing or migrating) and in-memory structures (which don't) rather than trying to freeze or migrate everything.
reply