Yes, the shared acronym makes Googling for anything AFL related rather annoying. The results are usually a mixture of Australian football, American rabbits, and fuzz testing.
Any idea as to why the following requirement? This will limit Afl.rs users quite a bit.
afl.rs needs to compile against a version of LLVM that matches rustc's.
The easy solution (if you can wait on a slow build) is to build rustc
from source and put it in your PATH. Then afl.rs's build script will
find llvm-config automatically. Otherwise, the environment variable
LLVM_CONFIG should hold the path to llvm-config when you build afl.rs.
I was under the impression that Afl can test any application that takes stdin. I'm underinformed for sure, so what's the idea behind explicitly adding code to support Afl fuzzing?
This might be working with libfuzzer, one of the tools afl lists on its homepage as derivatives/offshoots: http://llvm.org/docs/LibFuzzer.html
afl is fundamentally instrumentation-based and likes a good look at what sorts of different branches a program is taking, it's a little bit more hands-on than just feeding things to STDIN.
I think it injects code at branches to observe the branch coverage.
After all you have to compile your C programs with the afl-gcc binary.
Maybe you could ptrace every single instruction executed by the program and then stop
at every jump and use the relative values of the IP to identify branches...
I don't know if that would work, but it would be much slower.
Data input goes through stdin, but AFL needs to instrument the binary to observe branch coverage and changes in branch coverage across varying inputs.
From the alf-fuzz site:
> American fuzzy lop is a security-oriented fuzzer that employs a novel type of compile-time instrumentation and genetic algorithms to automatically discover clean, interesting test cases that trigger new internal states in the targeted binary.
I see, so does that mean it potentially would be required to explicitly support other code generators (GHC, OCaml, or something else), assuming the existing instrumentation is insufficient?
AFL is coverage-guided fuzz testing (fuzz testing that relies on code coverage).
Someone (not me) wrote an "LLVM pass" that loops through the generated LLVM IR for any LLVM-compiled program and injects the necessary AFL instrumentation such that AFL can work on it:
Since Rust uses LLVM when compiling a binary, all afl.rs does is incorporate this instrumentation and AFL just needs to be run on the resulting binary.
There are multiple versions of LLVM though. Rust is compiled using a specific version of LLVM. Since the LLVM pass above is reading the generated LLVM that Rust produces, the versions can't be too far off otherwise the LLVM pass might see some unrecognized symbols.
The README in the project suggests just compiling Rust which also compiles LLVM, then you'll have a version of LLVM that is guaranteed to be the same when you setup the instrumentation.
I think it might be sufficient to update the documentation to suggest the user of afl.rs just download and use LLVM 3.8 since this is the major version that Rust uses internally. I need to do more testing to confirm this will work for everyone, then I can update (and greatly simplify) the README.
Thanks, because the way which snapshots of LLVM and rustc are required to build a certain rustc release does not make it an easy to use solution for common Rust users. It'd be great if an existing LLVM 3.8 can be leveraged.
Note that with 1.10, the way we do rustc snapshots is changing: it will build with 1.9. And then 1.11 will build with 1.10, etc. so that will help a bit.
We are always interested in adding more platforms. I don't remember if there are any open issues with a MUSL host; I vaguely remember something about compiler plugins?
I suppose the issues you're referring to are with statically linked against musl executables. I wouldn't expect any such issue with musl just as the host, where you can run rustc on alpine and voidlinux, just not necessarily statically.
There's still a little more work to do though. Mainly, I want to compile AFL as a part of the workflow (see the afl-sys crate in the repo) so the user doesn't need to manually install AFL to use `cargo fuzz`.
Others here have explained the benefit - instrumenting. I will add: you do have the option of not instrumenting when using AFL, but instrumenting is a _lot_ faster - so much so that it is completely worth the extra effort. (It's actually not too much extra effort, either)
Randomly looking at the "Trophy Case", it looks like most of these errors are run-time "panics". We can break Rust errors down into three main categories:
1. Compile-time errors. This includes most memory-related errors, which are mostly caught by the borrow checker. These are very serious errors, often with ugly security consequences. Rust's big selling point is that it can catch many different kinds of errors at compile time—but not all.
2. Run-time panics. This includes "index out of bound" errors, integer overflow errors (in debug builds only), and assertions inserted by the programmer. This is Rust's second line of defense, so to speak.
3. Expected run-time errors. These are mostly reported using return values of type Error, which is the normal way to handle errors in Rust.
Most of the errors caught by AFL seem to be errors in group (2) that ought to be in group (3). In most cases, these errors couldn't be moved into group (1), because they're not the kind of thing that's easily caught at compile-time.
So this is a really cool tool for Rust developers, especially ones working on libraries that parse untrusted input. I was especially impressed by the fact that AFL could discover overflow errors, which Rust normally only protects against in Debug mode.
> These are mostly reported using return values of type Error
However, unlike more dynamic languages, the return types are usually Result<SuccessValue, SomeErrorType>, e.g. writing to a file or a socket will get you Result<usize, io::Error>, values of which can be Ok(usize) (the number of bytes written) or Err(io::Error) (an I/O error which you can further inspect).
Although if you want to, you can use Result<T, Box<Error>> where Error is a trait implemented by types representing some error information.
You then lose the ability to inspect errors other than for reporting them wholesale, but the APIs are slightly more uniform, type-wise.
> However, unlike more dynamic languages, the return types are usually Result<SuccessValue, SomeErrorType>
Yes, this is a more complete and precise explanation. :-) I rather like Rust's error handling in practice—it adds only a small amount of syntactic noise, and even the annoying bits (such as converting and wrapping error types) can mostly be isolated in a single file per project.
> 2. Run-time panics. This includes "index out of bound" errors, integer overflow errors (in debug builds only), and assertions inserted by the programmer. This is Rust's second line of defense, so to speak.
I still argue, to every Rust user I meet, that they should turn on overflow checking in Release builds too.
I strongly disagree with the idea that it's a Debug feature only. If it's not always on, then it's not actually a language feature. The performance hit, if you selectively opt-out for profiled critical paths (and thoroughly review them), is usually very small.
In my opinion the default should switch to "opt-out" instead of "opt-in" for Release builds.
> I still argue, to every Rust user I meet, that they should turn on overflow checking in Release builds too.
Yeah, I remember when the debug-mode overflow checks were added right before Rust 1.0. There wasn't enough time remaining to thoroughly measure the performance impact of these checks in release mode or to implement compiler optimizations to reduce the cost. So IIRC, the decision was made to allow integer overflows to panic, and to turn on checks in debug mode so that the ecosystem wouldn't come to depend on the absence of checks.
But AFAICT, the debug-mode checks are basically still a placeholder for some indefinite future version of Rust that wants to fix this issue properly, if the performance cost can be made low enough. And in the meantime, you can still turn the checks on for release builds if the performance hit is acceptable for your application. I'll consider your recommendation when building binaries in the future!
It's a fine line to walk. Performance is a deal breaker for a lot of people, whether it rightly should be of not. It doesn't matter how much safer Rust is if it's not used much.
If your goal is purely to make Rust as safe as possible, then you should advocate the view in your comment.
If your goal is to increase Rust adoption, and/or decrease the use of less safe languages for development in the future (in whichever prioritization of the two you like), then IMO the route described in ekidd's comment seems the quickest route there, IMO.
> If your goal is purely to make Rust as safe as possible, then you should advocate the view in your comment.
> If your goal is to increase Rust adoption, and/or decrease the use of less safe languages for development in the future (in whichever prioritization of the two you like), then IMO the route described in ekidd's comment seems the quickest route there, IMO.
My view is this is a false dilemma.
The "performance" case is not performance, because the benchmark is broken code. It does not work, because it is not secure/safe/reliable. It is incorrect to compare the performance of code compiled without overflow checking against code which is compiled with it enabled. They simply aren't the same program.
A correct performance comparison would be a program with manually inserted overflow checks everywhere, against a program with the overflow checks done automatically (by compiler).
Or, a performance comparison where code is selectively opted-out, where a critical path bottleneck is identified.
The same can be said of the performance comparison of "array index bounds checking". One program works, the other does not, so it isn't a fair comparison.
> A correct performance comparison would be a program with manually inserted overflow checks everywhere, against a program with the overflow checks done automatically (by compiler).
Correct is relative to the goals of the person assessing the comparison. If that person values performance over safety to enough of a degree, them a small performance loss will outweigh safety (regardless of whether you or I think this is a useful way to compare). If your goal is get people using Rust, and you believe people value performance, you can't ignore this and expect things to turn out how you like. This is the balancing act of a community and of advocates, making hard choices about what they would like to do, and what they feel is needed due to the constraints of reality.
> The same can be said of the performance comparison of "array index bounds checking". One program works, the other does not, so it isn't a fair comparison.
No, for some subset of array bounds checking, one works and the other doesn't. For the rest, they both work, because it isn't out of bounds. Whether you think people should program defensively and assume their code could fail, some will instead choose to assume they can deal with that as the programmer and choose to do so because it is more performant. You can either write those people off, or meet them half-way, and hope to escort them the rest of the way. Rust apparently chose the latter.
You can do it with either -C debug-assertions (stable) or -Z force-overflow-checks=on (unstable). The difference is that the former will also enable debug_assert! in your own code.
If you're parsing untrusted input, you should try to systemically avoid panics. Easier said than done, but I have started a small framework for protocol parsers called untrusted.rs that helps: https://github.com/briansmith/untrusted.
The main idea is that most panics (in parsers) are going to be caused by using the indexing operator on slices containing the untrusted input, so we wrap the input in a type that hides the slicing functionality, giving us only methods that return 'Result's.
It doesn't need to use llvm_mode. You can do what gcc does and have an afl-gcc binary which mucks with the assembly instead of actually inserting hooks smartly. This isn't as good, but works without nightly :)
AFL support for Rust is in fact better than that for gcc because Rust at least provides a plugin interface that doesn't require you to have its sources lying around -- the GCC support instruments the assembly instead of compiler IR, which is far less effective. Rust nightly at least has proper afl support, gcc doesn't.
It works for clang because llvm plugins in clang are stable, but we could get there easily -- in fact stabilizing something akin to `-Xclang` is probably something that could be done in Rust today without issue.
A stable version of afl-rs could exist that does the same thing gcc does, as I mentioned below.
As far as compiler tools are concerned, I'm mostly okay with stuff being nightly-only. These dependencies do not pollute the dependency graph and are something you only deal with locally.
You can write your own code in stable Rust while using tooling that works on nightly. Rustup.rs makes it very easy to switch between stable and nightly.
Fortunately for Rust that's not an issue since you don't need any special privileges to install it; hell, it doesn't even have to be installed. As long as you point your $PATH to Rust's bin directory you can use it, as far as I know.
If you work for a company that does not let you choose your compiler then I guess they probably will be stuck with a language more mainstream anyway. I doubt that the nightly compiler will be the dealbreaker.
On this companies using less mainstream languages like Swift, Go, F#, Haskell, OCaml isn't an issue because the communities don't depend on having nightlies installed.
> Every time someone posts a cool Rust library, most of the time it makes use of nightly.
Examples? Examples that aren't tools, that is. The main example I can think of is serde, and that works on stable (just works better on nightly).
What's wrong with nightly for using tooling? You don't have to keep updating, you can pin to a nightly from a particular date. You can use rustup.rs to switch to stable locally if you need to.
I said it's unfair, because gcc has worse issues here, and everyone probably agrees that gcc is "enterprise ready".
I'm not joking. Did you miss the part where I said the situation is worse for gcc? I do understand that this is a problem. I also do understand that it is not specific to Rust. Besides, convincing IT to allow rustup with an access-controlled toolchain directory shouldn't be a big deal in many cases. If they allowed one compiler, why not two? (Rustup can also be installed without sudo/admin, but I assume you are talking about situations where they want to restrict the executables running, not just globally installed executables). Yes, I know that in some cases they may not want two, but again, in those cases the GCC solution won't work either.
Note that tools like nvm etc are also quite necessary in their respective spheres for serious development. I was once in a situation where I couldn't even install node at a modern version without nvm.
For clippy, we are working on it, yes. For afl a simple fix exists (write the analog of -Xclang) but that fix will only work nicely if we ship llvm headers too which clang does (which we can, but it is a decision that can't be taken lightly). So no, we are not working on it. It is an interesting idea, and I will ping the relevant people (there are plans to ship "optional components" that you can obtain with rustup or download directly), but there are no concrete plans.
The general evolution of Rust is towards everything being done through rustup. If you want to install a cross compile toolchain, clippy, get stdlib sources, or a different compiler version, use rustup. All the solutions to the nightly problem involve rustup. I.e. instead of installing nightly and using clippy, you just `rustup install clippy` or something and the relevant clippy binary (or compatible source?) will be downloaded, one which will work even if you are on stable. Basically, rustup will be the thing you install when you want to use rust.
This is good, but as far as enterprise is concerned, you are still installing rustup, not rust. I'm not really sure what's wrong with this, but I'm sure some enterprise shops will have issues with this. For those which don't; the solution of having nightly installed via rustup in a frozen directory is not that much different, just unweildy.
I have worked in places where the computer is locked down for many teams. But not in a place where it would be
locked down in a way that rustc would be okay but rustup would not. I'd love to hear more about such cases and discuss how it can be addressed, but I'd prefer email (username@gmail or @mozilla.com). Thanks :)
> The general evolution of Rust is towards everything being done through rustup.
The general evolution of Perl was towards everything being done through cpan (the client). Decades ago. That doesn't stop companies from rolling their own Perl packages (when not just using what the system shipped with) and CPAN module packages (when it doesn't already exist as a package for the distro, or in a trusted third party repo, or is not the right version your code needs).
In many enterprise environments, when one or more production servers running your code fails, you want clueless intern #23 to be able to spin up a new box, running the default configuration for your distro, and pop the required packages on for your program, copy the relevant config or two that are needed for it to event start, start it up and expect it to "just work". Running rustup and remembering it needs to install a specific version to ensure everything turns out right is not conducive to that.
Now, this is less of a problem since rust is used to build other stand-alone programs, not necessarily run itself on as many servers, but it we're talking about a QA or testing box that dies, and there's a critical fix waiting to go out, I think it's obvious where this still has some importance.
You want your devs all running the exact same version of the compiler. The last thing you want is someone with a slightly different version than used by everyone else (and production!) which writes some code that compiles and runs fine without problems, but crashes or exhibits a bug in testing, QA, or worst case, production. Having something like that occasionally gum up your development pipeline is a good reason for pegging tool versions (depending on the type of tool).
Having things use rustup is great. Don't assume that's sufficient for certain populations of users (and a populations you really may want, such as enterprise). At best, they'll use rustup as a basis to start some other packaging, but even that's risky because rustup is complex, and it's hard to contain all the variables unless you're very careful about exactly what version you use with it, so then you're packaging rustup, and then package what it created. Easier at that point to just skip rustup.
It's all about reproducibility, which is king in enterprise. Rustup is about ease of use, and easy iteration. Enterprise already sacrifice that on the alter of reproducibility, so it's really not a selling point for them. It's great for everyone else, it just has less to offer for this subgroup.
My point is that it's not always a local tool. Sometimes what you consider a local tool might be something run on a QA or testing box, which needs to be tightly controlled to make sure that the difference between a tooling item intalled and a required library that is in production is closely tracked, and the tooling library needs to match exactly what's actually installed locally for devs. Think CI tool chains, automated testing, etc. A certain percentage of organizations where see "let's use this program that dynamically downloads and builds a program, which we can ask for a specific version if we want" as unacceptable because either it requires remote sources, it requires specific configuration which isn't tracked automatically or enforced, or both.
It's not about whether it's possible, it's about whether you're meeting the needs of the target audience. I don't believe rustup is meeting those needs in some cases. The important distinction is that possible does not imply acceptable.
I get that, but the solution is still possible and should be acceptable -- keep a One True Copy of a .multirust directory which you install on all testing servers and dev setups. No network necessary. Enterprise setups already do far more complicated things with pypi clones and whatnot.
Again, having rustup installed with two preinstalled and pinned compilers should not be much different, acceptability-wise, from having one preinstalled and pinned compiler.
I humbly suggest that saying something "should be acceptable" when it's been explained at length that, regardless of whether you think it's acceptable, other people do not think so, is exactly the problem yazaddaruvala was describing above. If you are trying to market to a population, you need to take their needs seriously with a good faith effort. Not understanding is fine. Doing a cost-benefit analysis and determining there's just no resources for what's needed is fine. Telling people that their needs are unjustified doesn't come across very well to those that still feel those needs.
I'm saying it should be acceptable in the parameters presented to me so far. Please explain why two compilers with a switching program is bad, whereas one compiler isn't. I am trying to understand the needs and come up with a solution, but so far they have been presented in bits an pieces. I am working off what has been told to me so far and my own experiences in locked down environments, which admittedly weren't as locked down as the ones you describe. I am also comparing with what other "enterprise-ready" languages require for tooling to try and ubderstand what is "acceptable". You keep telling me that I am not aware of the needs of enterprise users. Fine, I concede that. Please explain them to me. My intention in providing possible "acceptable" solutions is to find out what use case prevents them from working and see how Rust can address that. I am not telling people their needs are unjustified.
...this is also why I implored it to be further discussed in email, while the media are similar I am able to discuss more clearly there.
> Please explain why two compilers with a switching program is bad, whereas one compiler isn't.
Having any output that's based on a person remembering to set a configuration is less useful in these situations than having the configuration hard coded. (you don't want someone hunting down the correct config in some company wiki, much less working from memory. At most you want a config copied from a repository of configs).
Having a utility download a binary or source from the internet is less useful in these situations than serving it locally. (You can't control that the remote resource is still there, is the same, is secure).
Knowing that the exact same stuff that works in whatever development environment(s) you have (local, shared, both) works the same as when it's pushed to some further resource (automated build server, automated test harness, etc) is important in preventing bugs and problems.
In the environment I had experience with, we had a mainly Perl ecosystem running on CentOS servers. We created RPM packages for every perl module we pulled in as a dependency for our system or back-end RPC servers. Everything we installed on these boxes got an RPM built if at all possible. We maintained our own yum repository that we installed these packages from. While trivial to run cpan to install a module on these systems, that was not deemed acceptable for anything going into production. Rustup would not have been allowed into production here, nor on some of the shared dev and testing boxes we had, since that wouldn't lead to being able to build the exact same binaries easily and definitively. The absolute last think you want is a problem that loses data/config, and to find that you're not sure how to reproduce the last build environment exactly.
Does rust make assurance that things should build correctly with later versions? Yes. Does that really matter when you're talking about hundreds of thousands to millions of dollars? Not without a contract. That's one reason why enterprises use distributions that provide this feature, and build their own packages and deploy them where those distributions fall short.
> Having a utility download a binary or source from the internet is less useful in these situations than serving it locally
I already addressed that; I'm using rustup as a toolchain manager, not for downloading. You can have a .multirust directory that everyone uses and use rustup to switch. You could alternatively set rustup to download from a internal server that contains a reduced set of preapproved binaries only. I'm assuming that the external network is turned off here; it usually is in these situations. If you want a tool that manages toolchains but does not contain code that accesses the internet, that's a more stringent requirement that rustup doesn't satisfy; though I do hope Rust's distro packages work with update-alternatives.
You can also create an rpm for your specific two-compiler setup, of course. That's annoying though.
> Having any output that's based on a person remembering to set a configuration is less useful in these situations than having the configuration hard coded
Set default to stable, and `alias fuzz='rustup run nightly cargo afl'` :)
Again, this is for tooling, you can easily paper over the fact that the tool uses a different compiler.
Rust does have a tool (crater) that runs releases on the entire public ecosystem before releasing. There is an intention to make it possible for private code to take part in this too and provide feedback. But you're right, it is still unreasonable to expect enterprise users to trust that updates will work :) Any stability guarantee isn't enough, since bugs happen.
> I already addressed that; I'm using rustup as a toolchain manager, not for downloading.
I missed that, but see no problem with that.
> You could alternatively set rustup to download from a internal server that contains a reduced set of preapproved binaries only.
It's better, but less idea from a system maintainer's point of view than distro packages, because multiple package systems (which is essentially what rustup is in that use) is more work. It may be the best overall solution depending on how well multiple rust distro packages can be made to play together. Distro packages to provide the binaries and rustup (or something) to just switch between local binaries would be ideal for the system maintainer that wants control. I understand it's less ideal from the rust developer (as in someone that works on the rest ecosystem, as opposed to someone that works in the rust ecosystem), because the goals are different.
> If you want a tool that manages toolchains but does not contain code that accesses the internet, that's a more stringent requirement that rustup doesn't satisfy
Less that it can't (but that is a reality some places), more that it definitely won't, and someone exploring in it won't make it do so accidentally. Don't let the new dev accidentally muck up the toolchain.
> You can also create an rpm for your specific two-compiler setup, of course. That's annoying though.
Annoying for those wanting to get new rust versions out to people, annoying for devs that want the latest and greatest right away, but only slightly annoying, and easily amortized, by those that need to support the environment (ops, devops, sysadmin, whatever you want to name them).
> Set default to stable, and `alias fuzz='rustup run nightly cargo afl'` :)
If I was tasked on making sure we had some testing system in place that ran some extensive code tests in QA or something, or for a CI system, I wouldn't rely on that. It works, it's simple, but when it breaks, the parts I have to look at to figure out why and how, as all over the place, and deep.
Did someone change rustup on the system?
Did someone change the nightly that's used on the system?
Did someone muck up the .multirust?
If any of those happened, what was the state of the prior version? What nightly was used, what did the .multirust look like, did a new nightly catch a whole bunch more stuff that we care about, but aren't ready to deal with right now and is causing our CI system problems?
Theoretically I would build a $orgname-rust-afl RPM, and it would have a $ordname-rust-nightly RPM dependency. $orgname-rust-afl would provide a script that's in the path, called rust-afl-fuzz which runs against the rust-nightly compiler (directly, without rustup if possible. less components to break), to do the fuzzing. RPM installs are logged, the RPM itself is fully documented on exactly what it is and does, and after doing so once, all devs can easily add the repo to their own dev boxes and get the same setup definitively, and changing the RPM is fairly easy after it's been created. DEB packages shouldn't be much different, I don't expect other distros to be as well.
What did I get out of this? Peace of mind that almost no matter what happened if my pager went off at 9 PM (or 3 AM!) that weird system interactions, automatic updating, stupid config changes, etc weren't likely the cause of the problem, and if worse came to worse, I could redeploy a physical box using kickstart and our repos within an hour or two. When you have a pager strapped to you for a week or two at a time, that stuff matters a lot.
To achieve this we went so far as to develop a list of packages that yum wasn't allowed to automatically update (any service that box was responsible for) when everything else auto-updated, which were automatically reported to a mailing list as having updates so someone could go manually handle those by removing one of the redundant servers from the load balancing system at a time to update and restart the service (if not the server), re-join to the load balancer, and then move to the next server, for zero downtime updates.
The yum stuff was handled through a yum-autoupate system postrun script (a cron job script provided CentOS specific package). Yum auto-update didn't support this feature, so we added as a feature of that configuration, made our own RPM to supersede CentOS's version of the package, and then created another RPM for the actual postrun script to be dropped in place, and added them to our default install script (kickstart). We were able to drop support for our version of yum-autoupdate when CentOS accepted our patch.
All that's really just a long winded story to illustrate that sysadmins like their sleep. If your tool reduces the perceived reliability of our systems, expect some pushback. If your tooling works well with our engineering practices, or at least we can adapt it easily enough, expect us to fight for you. :)
Rustup is great, but when I was at this job, I would have had little-to-no use for it (besides maybe looking at how it works to figure out how to make an RPM, if one didn't exist to use or use as a base for our own). I know, because that's the situation perlbrew was in with us.
> Again, this is for tooling, you can easily paper over the fact that the tool uses a different compiler.
Sure, but in this scenario papering over is less important than easily discoverable and extremely stable.
> Distro packages to provide the binaries and rustup (or something) to just switch between local binaries would be ideal for the system maintainer that wants control
I think in this case update-alternatives or some such would be better? Not sure if it can be made to work with a complicated setup that includes cross toolchains. But I agree, in essence.
But anyway the local rustup repo thing was just an alternate solution, I prefer distributing the .multirust directory.
> Don't let the new dev accidentally muck up the toolchain.
> ...
> If I was tasked on making sure we had some testing system in place that ran some extensive code tests in QA or something, or for a CI system, I wouldn't rely on that. It works, it's simple, but when it breaks, the parts I have to look at to figure out why and how, as all over the place, and deep.
abstracting over rustup fixes this too. Keep .multirust sudo-controlled and readonly, don't make rustup directly usable, and just allow the tooling aliases.
> directly, without rustup if possible. less components to break
yeah, this is possible. It's pretty trivial to integrate afl-rs directly into a compiler build, perhaps as an option. You can then just build the stable branch of rust (so that you get backported patches) and use it.
> Rustup is great, but when I was at this job, I would have had little-to-no use for it
Right, as you have said there are other options out there to handle this issue :) Which is good enough.
When you do care about reproducibility but don't want to repackage Rust, rustup with a shared readonly multirust dir can be good enough. Otherwise, if you're willing to repackage rust, that works too :)
Sure, and to be clear, rustup works perfectly fine for my current needs. I just play around with it a bit when I have time, and even if I was to use rust in production, I would use rustup in my current environment (where the dev team consists of me, myself, and I ;) ). Almost all the benefits of controlling the packaging go right out the window when there's very few devs involved and they aren't expected to change.
First, I do want to say, I don't really care about the gcc alternative. I'm not even particularly interested in this tool. I've just found in the past, folks in the Rust community are quite dismissive about "other perspectives", particularly when they differ from "the open development way". I was just trying to urge you not to encourage that dismissive attitude.
Meanwhile, to potentially help you understand how, at least the company I work at functions:
Any time I need a library or tool, I need to:
1. Get approval from the open source team (for each minor version).
2. Start a security approval process (for each minor version).
3. Download the source, successfully modify the build scripts to use the internal build system and dependency management solution (for each minor version).
4. Upload the modified source to the internal dependency management solution (for each minor version).
5. Finish the security approval: Hopefully with a "Successful" ending (for each minor version).
Something like Rustup, in its current state, will not be usable (also its a bit redundant - we pin compiler/runtime versions in our BuildSystem.config). FYI, even a version of Cargo that can pull dependencies from a non-controlled repo, will not get approved (we would need to modify it to panic instead of call the network). Only a modified version which allows the internal build system to pull libraries from our internal dependency management solution, put the files in the correct dirs, will be allowed (fortunately I think Cargo is already able to do this). FWIW, this process would be almost identical to what we currently have to do for NodeJS projects and NPM packages.
Hopefully this gives more insight into why I thought it was funny, because:
1. Literally any "uncontrolled" tool that can talk to the network, download/build an executable, is entirely disallowed.
2. Getting approval for a single minor version is a pain. Requesting approval for a "nightly" i.e. unversioned is unimaginable. This would literally be the conversation:
What is the level of support on that version? Well, its not really versioned, so there is no level of support apart from upgrading to the latest nightly.
How would you update for a security fix? Update to the latest nightly.
Wouldn't that be an arbitrary number of commits, not just the security update, i.e. potentially the difference between minor/major versions? Yes.
So, we cannot minimally update the nightly for a security fix unless we cherry-pick the commit ourselves? Correct.
Also there is no support on the version of nightly we are pinned to? No.
So there may not even be a commit we can cherry-pick, we might have to investigate and make the change ourselves? Correct.
Also, there could potentially be a bug in our pinned version which only our company is exposed to? Well, its unlikely any other team/company will pin to that exact version, so yes, its unlikely we would even find out about a potential security vulnerability.
Ah, I see where you are coming from now. Thanks :)
> First, I do want to say, I don't really care about the gcc alternative. I'm not even particularly interested in this tool. I've just found in the past, folks in the Rust community are quite dismissive about "other perspectives", particularly when they differ from "the open development way". I was just trying to urge you not to encourage that dismissive attitude.
I'm usually not dismissive, since I've felt these pains too working in a (less locked down, but locked down nevertheless) similar environment. In this case I was dismissive because it is just as bad for GCC, which makes targeting Rust very unfair. Plugin tools like this always require a bunch of extra work up to even recompiling the compiler, which provides additional barriers to adoption.
--------
So, firstly, I wasn't advocating rustup as a tool which downloads the toolchains. For a reproducible build, that's not good. I was advocating it as a tool which manages toolchains and lets you switch with ease (you load it up with the toolchains you need beforehand, and distribute a .multirust directory to your build machines). This lets you use the supported stable version for actual builds, and nightly for peripheral tooling like linting and fuzzing, which doesn't affect the actual build -- just diagnostics. Note that your concerns about bugs don't apply to tooling, since it doesn't touch the outputted binaries. This may lessen the pain of getting approval, but still, a better solution should exist :)
If you are going to modify and build Rust sources, this isn't a problem anyway. You can get the sources for the latest stable (they're on a branch somewhere) and tweak them so that this tool works (one line change to code), or so that the compiler allows you to use unstable features (make/configure option). This is effectively a "nightly" build on stable sources, sources that will be getting backport fixes for bugs. Note that the APIs used by this tool will not be stable and may change (most likely they won't) if you update to a new compiler.
-----
The above solution is less viable for clippy, since clippy depends on tons of extremely unstable compiler internals (whereas this tool depends on one API call which could change but probably won't) which will break every compiler upgrade. But as I said, there is a fix being planned for clippy. For most users the fix will just involve using rustup. For enterprise users I imagine the possible fixes will either involve downloading the official clippy for their version of the stable compiler once, or, if you are compiling from sources, compiling Rust with clippy included (which probably won't be hard since building clippy will be part of Rust's rustup toolchain packaging anyway, so the steps will be the same and probably just a make/configure option).
> fortunately I think Cargo is already able to do this
Not exactly. Cargo so far is fine with local path dependencies, but it doesn't yet have support for having a locked down internal mirror repo with the relevant sources like many people do for PyPi or npm. That's something enterprise users do want, since vendoring in-tree is not always the solution. I don't know if it's being actively worked on, but it is on the list of things that Rust definitely wants.
> 1. Get approval from the open source team (for each minor version).
> 2. Start a security approval process (for each minor version).
> [etc...]
Out of curiosity, do you think having a laborious process to adopt even minor versions actually improves security?
Because to me, since in most software exploitable bugs are often fixed without being explicitly marked as such or CVE-ed, this seems like basically a recipe for insecurity.
I could be wrong. And I know that even if you disagreed with the process, that wouldn't mean that you didn't have to follow it, or that many, many developers in a multitude of environments similar to yours don't have to follow similar processes. But I'm curious what you think.
That sounds like a horrible work environment. Seriously, devs can't choose their favorite tools, or specifically use tools which improve their quality until IT approves it?
I've worked in bad places where it was hard to upgrade to the latest version of a language, but at least that was always under full control of the Dev group.
Would you really want every developer installing their favourite tools/libs and having a complete free-for-all when you've got 200+ devs, each with their own opinions and preferences?
IT have to support this stuff, they don't want to be rebuilding the software stack every six months or so.
This is not about a tool to provides a syntax, or makes commits. It's a tool that runs an instrumented version, only on the location the dev specifies, and reports actual bugs. If your team has a difference of opinion on whether you should fix actual bugs, you have larger problems.
That said, if your program only feasibly runs in a shared QA/dev environment, then yes, it can be understandable to prevent devs installing random tools and utilities there. But in that case, the problem is not so much hedonistic developers, but a lack of separating portions of the code base into individually testable components. Again, you probably have larger problems than whether your devs can install their own tooling (because it's of limited use in a lot of cases anyway).
To be clear, there are good reasons for restricting certain classes of tooling. You don't want the absent minded dev installing some cloud based source code integration tool. Then again, the only way to really protect against this is to isolate the dev network, and not allow internet access. I imagine the loss in productivity is fairly great. I'm sure there's a happy (or not!) medium in there, individual to most companies.
Because a lot of companies, at least those that develop their own internal software (on Linux for example) often stick to the compiler the system ships with, e.g. CentOS, say.
GCC is in that respect "enterprise ready" because it very likely was installed on the system already. And probably not the latest version either.
In rusts case it sounds like we will ship rustup.rs in distros, since it is also intended to be how people obtain cross compilers and whatnot. I could be wrong about this -- I am aware of, but not directly involved in, the plans for rustup. If not via rustup, I would expect everything that can be fetched via rustup to be in the package repo. This solves the problem for clippy but not for afl, though there are solutions there too.
The GCC plugin for afl requires you to find the right GCC sources and compile against those (unless you use the hacky, less-awesome assembly based afl). This solution is available to you for afl-rs too, just not documented because using a nightly is easier. This makes afl-rs just as bad as GCC in this aspect. afl-rs needs a nightly not because it only works on specific versions, but because it hooks into the compiler in a way that is only allowed on nightly. Thus it will work on any nightly, even that from a few weeks ago, but it must be a nightly. As I mentioned it is possible to get it to work if you are willing to do what GCC does.
> In rusts case it sounds like we will ship rustup.rs in distros
I sincerely doubt any enterprise level distribution will make that the primary source of obtaining a rust compiler. I would bet good money that Red Hat won't. It's counter to most of the reason enterprise distributions exist in the first place, which is a stable (often at the cost of many other feature) base on which to build and support. If they aren't doing this, they aren't an enterprise distribution. This goes as far as Red Hat back-porting bug-fixes to 7 year old versions of utilities and libraries, because any change .needs to be backwards compatible at all cost. Additionally, there's the question of ensuring that all relevant portions of the rust install are removed on package removal.
At the same time, getting the source and every patch applied to it, and the complete compile instruction for a package is generally very easy. It's meant to be repeatable. For Red Hat, you just get the appropriate srpm instead of rpm, and that unpacks a source tarball and any associated patches separated, along with an rpmbuild config that contains all the info used to build an rpm package from them. Every single thing, including the kernel is built this way.
That's not to say rustup won't possibly be shipped as an extra utility, in one of th emain rust packages or as a separate one, but it definitely won't be the primary method most distributions provide for rust development (in the case of the non-enterprise distros, because it's just not as convenient as saying "install rust" and having it just be there, in a confirmed tested version that works with whatever other packaged tooling they provide. There are downsides to this, but there are also many benefits.
Right, which is why I said for such distros I expect rustup-obtainable extras to be packaged as well and installed in the same way. Again, unsure if this is the actual plan.
I'm not sure where you said that, so I'm not sure what context to interpret that in. Does "rustup-obtainable extras to be packaged as well and installed in the same way" mean packaged through distro specific packaging mechanism, or something else?
My only point in this part of the thread, which has little to do with the "we use gcc because it's stable" argument, is just that you shouldn't expect the distros the change how they ship just for rust, which expecting them to use rustup would be (in what I think are the majority of cases). If rust as a community expects rustup to be what distros use, then I think there's a disconnect from reality happening, and it's worth looking at because if that's important to rust (which I don't know, honestly), then it should be addressed. I understand that you may not know if this is the actual plan.
Note: I don't particularly think rust requirly a nightly build for this is important. Organizations will look at it and assess it as a tool as needed, and if they think it's worth using and they want to roll it out, they'll package it in some manner if that's important to them (and that might just be a tarball, at the simpler level), or they won't if it's not. Stability is important, but at the tooling level, it's much less so, and being able to make sure all yours devs are in lockstep is probably more important (for large orgs).
> "rustup-obtainable extras to be packaged as well and installed in the same way" mean packaged through distro specific packaging mechanism, or something else?
yep. I was referring to "If not via rustup, I would expect everything that can be fetched via rustup to be in the package repo."
What I meant was: instead of users installing rustup and then installing all the toolchains through rustup (`rustup install stable`, `rustup install arm-linux-gnueabi`, `rustup install clippy`), they can also install `rustc`, `rustc-arm-linux-androideabi` (cross toolchain), `rust-clippy` through the package manager (apt-get install or whatever). Cross compile toolchains for C are often similarly packaged (gcc-cross, etc).
The way I understand it (again, unofficial, not sure if it's the actual plan :) ), we would like most users to use rustup as the single source of truth for Rust. This won't work for some setups, including distro packaging, and I'm not sure what the plan is there, but I'd expect it to be something similar to what I wrote above. The binaries that rustup downloads will always be available, regardless, if you are okay with downloading a signed binary once and internally distributing it. But I don't expect it will come to that. Though right now Rust hasn't yet gotten all the ducks in a row for proper distro packaging, so that's all you can use. This is being actively worked on, and IIRC all the major work is done (?)
That basically matches with what I expect, but I'm not sure that distros will even bother to package rustup in the base packages (as something that provides an alternative to a packaged item, not just a supplement, such as cargo). In this respect, the wants and needs of distros are often at odds with those of developers. Developers want the latest with all the features in front of their users, distros want something easy to maintain and support (which often means not changing, or on a predictable schedule). This is particularly hard on interpreted languages, as the version shipped matters a lot more for what will run, not just what will compile in your dev environment. There's a (relatively) long history here.
If rustup isn't part of the base (distro-provided) packages, it will definitely be part of some popular add-on repos though.
I was super annoyed when I found out Ubuntu ships an ancient node and nvm doesn't exist in the distro. I would love for rustup to exist in the distro, but we should (and probably are doing so) plan for cases without it.
In some cases, all you have to do is the minimal amount to make it easier on the distro users, and people will jump on it. That may be providing repos for debian and red hat based systems where you provide a rustup RPM. That takes care of both making it easy (if not quite as easy as piping a remote source to a shell), and more secure. The keys are cached when the repo is added or first accessed, so if bad rustup packages are uploaded at some later time, the package managers will complain, loudly, at least for people who already had it, which is better than nothing.
I think Rust is interested in doing more than the minimal amount, and has been discussing with maintainers already, but I'm not aware of the details. They probably can be found on threads on internals.rust-lang.org or users.rust-lang.org though.
rustup.rs already does check against Rust release signing keys, so that part is at least trivial to integrate.
> rustup.rs already does check against Rust release signing keys, so that part is at least trivial to integrate.
I saw that (which is good!), but that's actually a step removed from what I was referring to. If a repo provides rustup, the rustup package itself is verified, which is useful way to get rustup itself compared to the fairly prominent and obvious suggestion at Rust's download page[1], which is "$ curl -sSf https://static.rust-lang.org/rustup.sh | sh". Targeting rustup.sh itself would actually be a fairly effective attack, since it's the tool that does the download, that verifies the binary, and that is often (I assume) run in a transient manner, since it's suggested you pipe the web response directly to an interpreter. If it was attached successfully, we would all be lucky if all they did was do nefarious stuff directly from rustup.sh, and not try to install their own,slightly changed, builds of the compiler[2].
In this particular instance, there's no reason to be shipping afl.rs in production code. It's just a tool that's meant for local development, testing environments, or maybe even part of a CI workflow.
I think this is a legit complaint. I think I understand why the rust team wants folks to use nightlies for unstable features. But if you had to opt-in with special cargo/compiler flags then it would probably be sufficient.
(Such a flag exists as an environment var, but you're not supposed to use it)
I guess the idea is to dissuade people from using hacks to enable unstable features -- this will break things which depend on the library. If a library compiles on stable, it should never break due to a compiler upgrade.
(Tooling doesn't have this problem, it doesn't come into play when it's used on a dependency)
The point is that your serious enterprise customers who can't change their toolchain also can't use AFL in GCC, so blaming Rust for this makes no sense. If anyone's to blame, it's AFL for needing custom compiler instrumentation. (Though I think that's a pretty weak criticism anyway.)
(Technically they can use AFL in gcc, just the less powerful version --which rust stable can too, if frewscxv adds it :). I'm not sure if a more powerful version for gcc exists, like it does for clang and rust)
There was seemingly no interest at all in upstreaming this when I asked the afl-dev mailing list. My hope was that along with afl-clang-fast it could, in time, completely subsume the hacky `afl-gcc` and `afl-clang` scripts, making afl much faster, more robust and more portable out of the box (I initially tested on POWER8 machines). This component is under GPLv3, so it might possibly be due to licensing concerns. My other changes were accepted upstream (posted at a similar time), so that's just a speculative guess.
People are seemingly interested in this and I've seen this question pop up a few times since authoring the code originally. Perhaps it is worth maintaining a fork or patchset for wider availability... The above code works, but is less efficient than `afl-clang-fast` and strictly proof of concept. Neither of those are insurmountable. In fact because this code works on GIMPLE very directly, I see no reason why it couldn't work for things like gccgo or gcj, or even GNU Ada or FORTRAN, I suppose, with some tweaks.
Interestingly, a team at Oracle seems to have (at a similar, or later time) reinvented this same thing when they attempted to fuzz the Linux kernel with afl. But that source code doesn't exist publicly, it seems. Mine may not have even been suitable for them, even if they knew of it beforehand.
reply