pouët.net

Rust and demomaking

category: code [glöplog]
Rust started to be somewhat popular in the demoscene.

It is of public knowledge that Rust is used by Ferris for most of Logicoma demos ( see his presentation of their demotool Demotivation ) and also by cpdt of Monad for ( at least ) their synth ( see Axiom ).

In the spirit of the Tiny Intro Toolbox Thread, I would like to start a thread where we could share our knowledge and resources about Rust, especially in the specific context of the Demoscene.

All contribution welcome :) !
added on the 2019-04-24 10:58:48 by maeln maeln
I have a first question: For people making size limited demos with Rust, how are you dealing with #![no_std] ?

In my experience, it has been a big pain: Only available in nigthly (although that may have changes), recurrent breaking change, and things like #![feature(start)] not being supported on various platform.

But since some of you made it work, I am curious on how you did it.
added on the 2019-04-24 11:04:03 by maeln maeln
I'm trying to do something, really really slowly, using wasm + rust, ideally I want to use most of the same code to target both the web and desktops (windows most likely).
.
added on the 2019-04-24 13:27:26 by xernobyl xernobyl
It's a pretty good language. Better than C++, worse than C... Fucking awful learning curve, at least for me, but I'm having tons of fun with it, it kind of forces you to write better code, instead of crappy single use code.
added on the 2019-04-24 13:29:31 by xernobyl xernobyl
First off, I believe blueberry and emoon are also using Rust extensively. I'm sure there are many others.

Second, The bulk of our code is in the tooling, which doesn't have any of these concerns (minus the engine/JIT compiler which is a crate that both the intro replayer and tool use, but that's actually somewhat small/straightforward). For the stuff the replayer uses, we build everything on nightly and use no_std and some core. Generally this hasn't caused _that_ much pain; we have our own Vec and fill in some iterator stuff, we use a custom allocator (both in the intro and in the tool), we do the typical lang_items shims, etc. We only target Windows with this code (we used to target mac as well for the parts that were used in the tool, but this required pretty minimal stuff and I dropped it when we moved to OpenGL 4.3 for compute). We have of course run into times where bootstrapping and interfaces have changed from beneath us; typically we've handled this by documenting the specific rust version we build with and stick with that for a bit (usually until an intro is finished) and then upgrade afterwards. I believe we did this 2-3 times; things seem to get more and more stable though, and our binaries get smaller and smaller (not accounting what we get from our packer, which is also rust [with some asm], but doesn't need to care about this stuff ofc).

Quote:
worse than C
I agree with this only insofar as FFI/bindings are concerned, i.e. interfacing with other C code and/or hardware directly. In my experience it's generally far better than C/C++ (our core tool/compiler code is really very straightforward, especially the threaded parts), but I'll admit having the replayer/engine parts in C (which interface OpenGL more than anything else) is a bit verbose in some places (though it's nice to have the same engine code in both places, rather than a separate replayer beyond just some bootstrapping).

It's difficult to say "we did it like this" and list everything exhaustively as our codebase has grown pretty organically and side-by-side with Rust over the past 3 years (or so), so beyond releasing source (which I'd like to do but won't happen any time soon) I can't really give a detailed list, but I'm happy to answer specific questions if you have them.
added on the 2019-04-24 13:42:50 by ferris ferris
Quote:
having the replayer/engine parts in C
should read "having the replayer/engine parts in Rust instead of C"
added on the 2019-04-24 13:43:48 by ferris ferris
Probably also worth mentioning we have our own Win32/GL bindings. These are also pretty straightforward (essentially just function lists and copied constants), albeit somewhat verbose. In some sense though I like to know _exactly_ what we're pulling in, so I can't really complain about that :)
added on the 2019-04-24 13:46:19 by ferris ferris
I consider cleaner syntax to write more reusable code very subjective argument to fav Rust.

Learning curve may be confusing and that's the main reason why I dropped it so far to focus on clean and reusable, modern C++ code myself.

I see from above that it may be interesting for small exec/intro coding.

For this use-case is there any particular advantage provided by the platform or it's all driven by individual experience?
added on the 2019-04-24 14:21:29 by hollowone hollowone
My thing with C has to do with the simplicity of the language, it's easy to know pretty well what all the language features do, while some things in Rust are not so transparent, like knowing that Option<Box<T>> actually translates to a single pointer. And it's hard to force Rust to do some things, when it assumes that whatever you're doing will cause an error, race conditions, pointer aliasing, or whatever, but that might have to do with my lack of experience in the language, considering that I first started compiling C code more than 15 years ago... I kind of gave up learning "modern C++" a while ago, since I'm not using it professionally for a while now, and I'm hoping to use Rust professionally at some point. Hopefully next year if we get a series A and don't go to startup heaven.
added on the 2019-04-24 16:02:13 by xernobyl xernobyl
For now, I do agree that the simplicity of C make it attractive to do the "low-level" stuff (like calling win32 api). But for the rest of the code, I find that Rust free you from a lot of cognitive overhead. I especially prefer the Haskell-like trait system. Templating and OOP in C++ is a living nightmare for me.

Hey Ferris :) .

Quote:
so beyond releasing source (which I'd like to do but won't happen any time soon)


That's very nice, I am very curious about a lot of aspect of Demotivation, glancing over the source one day would be awesome !

By the way, how does your custom allocator works ? Is it similar to something like wee_alloc ?

I noticed you had a
Code:get_proc_address.rs
in your tiny-gl crates. Is it full Rust ? I did a crate to do this but it's basically a wrapper to the specific part of glad that do the pointer loading (gl_loader).

Doing a JIT compiler in Rust is not too complicated ? It glanced over a few implementation and it was quite scary x) .
added on the 2019-04-24 19:42:57 by maeln maeln
Quote:

Templating and OOP in C++ is a living nightmare for me.


Practically speaking, where do you need templating so much that you cannot avoid it today with a different pattern considered as a best practice?

I personally don't do templating too much these days.
Not needed beside some generic and totally not optimized cpu version of the core math with vectors and matrices in it..

But then I realize that my default precision for instanciating them is the only compile time implementation I ever use...
added on the 2019-04-24 20:16:14 by hollowone hollowone
Quote:
I consider cleaner syntax to write more reusable code very subjective argument to fav Rust.

Learning curve may be confusing and that's the main reason why I dropped it so far to focus on clean and reusable, modern C++ code myself.


I think it's certainly subjective as well. Rust matches very well the way I like to think/simplify things; there's a reason why I picked it up in the first place and generally I've felt very little friction with the language. This is totally ymmv.

Quote:
I see from above that it may be interesting for small exec/intro coding.

For this use-case is there any particular advantage provided by the platform or it's all driven by individual experience?


I think this is a bit of a mix. For the actual small parts of the intros we do, I'd argue Rust is generally slightly _worse_ than C, as what we're doing is fairly sane/small and it gets a bit tiring writing `unsafe` everywhere perhaps. But it's not _that_ bad either, and results in the ability to use the same engine code in the intro and in the tool. And I think we get a _huge_ benefit in terms of the tool code, especially with our compiler, threaded code (mainly for file watching/asset reloading), data model, and the ability to pull in/manage external libraries (which we use for font/model loading, project [de]serialization, clipboard interaction, parsing, ...). For UI, we expose a C API to the core data model code, and consume that from a minimal Qt/C++ layer, and this works extremely well, though it can be somewhat verbose. We never have crashes, memleaks, and rarely bugs (when we do it's typical OpenGL state leaks or user errors with the JIT, which doesn't protect us from unlimited recursion by design; nothing that breaks our projects or anything), everything is fast and snappy. Generally I'm very happy with this.

Keep in mind that I don't really care who uses Rust or anything, so if it sounds like I'm trying to sell something, please ignore that and insert appropriate grains of salt where necessary. I'm just very satisfied with our approach. :)

For some other questions...

Quote:
But for the rest of the code, I find that Rust free you from a lot of cognitive overhead. I especially prefer the Haskell-like trait system.


Yes, this is very nice, although since our data model is relatively straightforward I actually don't think we use many of these features very often (and when we do it's in the tooling code).

The big exception is `Drop` ofc; we use RAII in a _lot_ of places to make GL state leaks a lot less common, and to eliminate a lot of manual memory management.

Quote:
By the way, how does your custom allocator works ? Is it similar to something like wee_alloc ?


The custom allocator is _very_ simple. At the lowest level we have `malloc`/`free`/`calloc` implemented on top of `HeapAlloc` and `HeapFree`, and some aligned versions of these that we implement on top of those (mainly to please SIMD code). We use these internally to implement various collections in the engine directly. Of course in a 64k we're probably being a bit more conservative than we have to be here (though I'm starting to suspect we can just impl `free` with `unreachable_unchecked` and maybe the compiler works its magic and saves a ton of space for us... hehe).

Anyways, that covers code that needs those functions directly - to make Rust use the same allocator, we implement the `GlobalAlloc` trait as such:


Code:use tiny_alloc::*; use core::alloc::{GlobalAlloc, Layout}; pub struct HeapAllocator; unsafe impl GlobalAlloc for HeapAllocator { unsafe fn alloc(&self, layout: Layout) -> *mut u8 { malloc_aligned(layout.align(), layout.size()) as *mut _ } unsafe fn dealloc(&self, ptr: *mut u8, _layout: Layout) { free_aligned(ptr as *mut _); } }


..and in the intro and in the tool, we use the following code to instantiate it and make Rust use it:

Code:#[global_allocator] static ALLOCATOR: HeapAllocator = HeapAllocator;


That's literally the entire thing (minus the impl of `malloc_aligned` and `free_aligned` ofc, but I'm sure you can figure those out).

Quote:
I noticed you had a `get_proc_address.rs` in your tiny-gl crates.


Code:use tiny_common::*; use tiny_alloc::*; use tiny_wgl::*; use core::mem; pub fn get_proc_address(symbol: &str) -> *const Void { unsafe { let len = symbol.len(); let symbol_ptr = symbol.as_ptr(); let ptr: *mut u8 = mem::transmute(malloc(len + 1)); for i in 0..len { *ptr.offset(i as _) = *symbol_ptr.offset(i as _); } *ptr.offset(len as _) = 0; let ret = wglGetProcAddress(ptr as *const _) as *const _; free(mem::transmute(ptr)); ret } }
:) (this can surely be cleaned up a bit)

Quote:
Doing a JIT compiler in Rust is not too complicated ? It glanced over a few implementation and it was quite scary x) .


Nah, ours is not that bad. See this for some early tests (and even how we used to make it work for mac too). It's more or less the same still (which reminds me, need to optimize that some more too, but it seems quite small/packable already...)
added on the 2019-04-25 14:02:21 by ferris ferris
Should probably mention that `tiny_alloc` referred to above is our internal crate which exposes `malloc`/`free` etc as I described above, and that's it.
added on the 2019-04-25 14:06:39 by ferris ferris
Also bear in mind that a JIT is only as complicated as the opcodes you're JIT'ing; ours stays relatively simple because they only do a handful of different kinds of things. We only need to support a very basic "language" to manage the CPU-side GL stuff and glue together engine bits/assets/sync tracks. So it's not a huge leap from a simple interpreter (and I don't really have a great reason for making that leap either; it sounded fun at the time and totally was, so).
added on the 2019-04-25 14:11:28 by ferris ferris
Quote:
Keep in mind that I don't really care who uses Rust or anything, so if it sounds like I'm trying to sell something, please ignore that and insert appropriate grains of salt where necessary. I'm just very satisfied with our approach. :)


It's all right. I liked your comments. Thanks for sharing.
added on the 2019-04-25 23:35:49 by hollowone hollowone
I wrote an 8k intro for linux in rust. I have the source available here if people want to see my no_std setup and the like: https://github.com/aleksanb/fourkay. It’s a real nice language to write in, although juggling raw pointers can be a tad difficult at first when you’re trying to use C APIs that expect pointers to uninitialized variables on your stack and rust really doesn’t want you to do that stuff. I’d never be able to write this stuff in C without segfault galore though, so that’s another advantage with rust for me.
The VST plugins for Oidos and Cinter are written in Rust, using the vst crate. An interesting feature of this crate (which I was also heavily involved in) is that it is structured in such a way that it is impossible, thanks to the Rust type system, to run into data races even when the VST host calls methods in your plugin from multiple threads concurrently.

For many years, I longed for a good systems programming language with modern syntax, properly well-defined semantics and a strong, expressive type system. Rust came along and was exactly that, and it was even better than I could have imagined.

I would say that Rust (or at least Safe Rust) has an honest learning curve. It has one extremely complex feature (the lifetime system / borrow checker), one moderately complex feature (the trait-based type system), and the rest are actually pretty straightforward. And after the compiler has accepted your code, there are usually very few surprises.

In contrast, C++ is deceptive in its apparent agreeableness. You can quickly learn to use it, and then you can continue learning the language for decades and still run into syntax you don't understand and runtime gotchas you didn't know you had to be aware of.
added on the 2019-04-27 19:01:59 by Blueberry Blueberry
Quote:
For many years, I longed for a good systems programming language with modern syntax, properly well-defined semantics and a strong, expressive type system. Rust came along and was exactly that, and it was even better than I could have imagined.

I would say that Rust (or at least Safe Rust) has an honest learning curve. It has one extremely complex feature (the lifetime system / borrow checker), one moderately complex feature (the trait-based type system), and the rest are actually pretty straightforward. And after the compiler has accepted your code, there are usually very few surprises.

In contrast, C++ is deceptive in its apparent agreeableness. You can quickly learn to use it, and then you can continue learning the language for decades and still run into syntax you don't understand and runtime gotchas you didn't know you had to be aware of.


All of this <3
added on the 2019-04-27 21:36:56 by ferris ferris
Holly molly thank you a lot Ferris <3 .
added on the 2019-04-27 22:06:09 by maeln maeln
Quote:
Holly molly thank you a lot Ferris <3 .
<3
added on the 2019-04-28 16:35:32 by ferris ferris
Exactly what Blueberry said. For our Amiga stuff we don't use Rust for the run-time (Amiga side) but I use it quite a bit for various tools, prototypes and such.
added on the 2019-04-28 18:13:31 by emoon emoon
Quote:

In contrast, C++ is deceptive in its apparent agreeableness. You can quickly learn to use it, and then you can continue learning the language for decades and still run into syntax you don't understand and runtime gotchas you didn't know you had to be aware of.


RUST looks to me like great exotic niche with a heavy competition in the market, I'm curious about it - for a hobby project, yes - but I would not deprecate C/C++ for it.

Back in the historic times I considered C++ very much as compiler driven experience. At basic level it was all the same, but then all implementations were completely different dev platforms with different tactics to get most from it.

That was applied to tools/compiler itself, language features and most critical standard library, that was a fiction. Higher level framework like those for GUI were also numerous to prove immaturity of the market and limitation of the system libraries.

For demoscene/low level related use cases I ended up creating my own std libraries for most of the things in DOS to maintain control and I was wrapping like crazy in Windows to achieve the same.

But this is a story from 95-2005..

Then I forgot about C++ for 10-15 years in favour to various managed platforms, then to rediscover C++11/14/17/20.. modern approach, quite recently.

And I must say, it's very satisfactory and finally quite unified and consistent experience.
But I also agree it's for experienced developers that understand how to maintain simplicity in the code base.
added on the 2019-04-29 11:52:22 by hollowone hollowone
C++1x != simplicity in code base
added on the 2019-04-29 13:06:28 by arm1n arm1n
Quote:
First off, I believe blueberry and emoon are also using Rust extensively. I'm sure there are many others.


:wave:
added on the 2019-05-14 14:21:57 by phaazon phaazon
I have been observing and even experimenting Rust, but for now C++ is still my default choice. Might or might not change in a few years.
added on the 2019-05-15 06:24:55 by imerso imerso

login