[SOLVED] Setting up network at boot without systemd

sighlinux · 2026-07-08 20:05:15

I have set up an Archlinux computer with llama-server.
There is just one thing I am not getting right: setting up network at reboot.
So I have configured the system start using cron @reboot calling my starter script.

The network setup script that gets called from the main starter script is still somewhat wrong:

ip link set enp2s0 up
ip link set eno1 up
ip addr add 10.10.20.33/24 dev eno1
ip route add 10.10.20.1 dev eno1
ip route add default via 10.10.20.1
ip link set enp3s0 up
ip link set eno2 up
ip addr add 10.10.40.33/24 dev eno2
ip route add 10.10.40.1 dev eno2
ip route add 10.10.40.0/24 via 10.10.40.1

I have to call the script twice manually after boot to make network work.
What did I wrong, what do I need to change?

(The multiple "ip link set ... up" probably aren't necessary... network things are so complicated on non-Debian linuxes since ifconfig has been removed )
(Yes I know, as BSD user I should have used Devuan. But I wanted to give Arch a try. Please forgive me.)

Is it possible at all to avoid using systemd or "network managers" for this?

Last edited by sighlinux (2026-07-10 15:58:22)

seth · 2026-07-08 20:22:30

You won't need some network manager but every task (incl. crond) will be started by systemd directly or indirectly.

I have to call the script twice manually after boot to make network work.
What did I wrong, what do I need to change?

Hard to tell - either the NICs aren't there/responding or there's maybe interference from an actually running network service?
(networkd is enabled by default, I think)

Why are you setting enp*s* up? (assuming they're just aliases for the eno* NICS)

sighlinux · 2026-07-08 21:06:22

seth wrote:

You won't need some network manager but every task (incl. crond) will be started by systemd directly or indirectly.

Yes I know, I just would like to avoid to touch anything systemd... *afraid*

seth wrote:

Hard to tell - either the NICs aren't there/responding or there's maybe interference from an actually running network service?
(networkd is enabled by default, I think)

how do I find this out? The NICs are just ordinary i350, nothing exoticc.
ps axuww|grep netw doesnt show anything, and systemctl list-units only shows systemd-resolved, nothing else what I associate with network. I also added systemctl list-units >/root/units_list at the start of my script, the output is the same, no networkd or anything like that.
The system is very basic, just the Arch base system, Nvidia drivers+CUDA and llama-server.

seth wrote:

Why are you setting enp*s* up? (assuming they're just aliases for the eno* NICS)

I read somewhere that this would be necessary, but it didn't change anything. ip a shows two altnames, the enp*s* name which iirc debian uses as main name, and enx<MAC>. Removing this extra ip up didn't change anything... any idea?

Last edited by sighlinux (2026-07-08 21:08:45)

seth · 2026-07-08 21:10:37

how do I find this out? The NICs are just ordinary i350, nothing exoticc.

Check "ip a; ip r" after the script runs to see what failed and of course also pay attention to which command maybe issues an error - do you run the script automatically or manually?

sighlinux · 2026-07-09 07:50:59

It turned out that the very last command fails, giving a strange error message "RTNETLINK: File exists"

When modifying the script I noticed that the modifications get ignored.
This gave me the impression that maybe something caches a previous version of the script file.
set +h didn't change anything.

So I tried syncing and purging the caches by sync;echo 3 >/proc/somesickcache.
After that the system broke down.

Now the scripts do no longer get executed at all by cron at system start.
Cron now just says "Bad command". (/etc/crontab contains only one line: "@reboot /root/systemstart.bash"...)
Parallel to that, I see in journalctl that crond now croaks directly after that, stating that getpwnam() failed. I do not understand this. I believed the stuff before going multi-user would always be executed as root. (There are no user accounts, I log in always as root.) (Regarding user accounts, in /etc/passwd there is a nologin user named "systemd-network" - maybe this is what interferes??)

Unfortunately Linux seems to have no boot environments, so I cannot simply roll back to a defined previous version.
Now I ask myself desperately, is it worth all the effort, or wouldn' it be better to retry using Devuan. (or even debian, as it uses a defined file-based network configuration so one does not need to touch systemd things) :cry:

Update:
At least the script works if started manually after booting. The failing last command seems not absolutely necessary if the clients access the llama-server from the same 10.10.40. network. So, for now I commented it out.
So the issues remaining that still need to resolved are these:

1. Find out why changes in the script do not instantly get respected. If I change something, and immediately afterwards execute the script, the output shows that which gets executed by bash is the previous version of the script, which has been overwritten on disk by the modified version. Only after some time passed and after repeated running, suddenly bash seems to re-read the script file. So there seems to be a cache issue. There must be a bash cache file on disk, because even after reboot the previous, physically no longer existing old version gets executed instead of the actual one.
2. Find out why crontab @reboot does no longer work.

As the project is still in the experimental stage, it is only annoying but no real problem to have to start the configuration scripts manually after reboots. So there is no hurry to find out what is going wrong. Sigh.

Last edited by sighlinux (2026-07-09 09:06:28)

Lone_Wolf · 2026-07-09 09:45:55

$ pacman -F ifconfig
core/net-tools 2.10-3
    usr/bin/ifconfig
$

ifconfig is still available on archlinux in the package net-tools .

Note that on archlinux systemd is also PID 1, if you want to use another init system you may want to check https://artixlinux.org/ .

seth · 2026-07-09 15:11:03

I suspect that either cron already runs in the initramfs or that /root gets mounted after cron starts or the script isn't executable.
Dropping the file caches will however have absolutely no impact on anything after a reboot.

Please post your complete system journal for the boot:

sudo journalctl -b | curl -s -H "Accept: application/json, */*" --upload-file - 'https://paste.c-net.org/'

sighlinux · 2026-07-09 19:19:01

@seth
I guess you are right. Otherwise I just do not understand the behavior. Just posted the journal as you instructed. Thank you very very much for looking into that!

I am sure I must have been done something bad what broke it. Maybe it was that "systemctl cronie enable" I tried when I found in the journalctl the repeated message that cronie is disabled. I do not know what is in the init ramdisk image... on the ext4 filesystem all my scripts mentioned have 754 permissions. When I run the scripts manually after boot, they work, ip a/ip r output looks fine to me.

@Lone_Wolf
For me, the init system is not that important. I just need something that is easy. Making systemd units is not at all easy for me. That is the reason why I tried the crontab @reboot idea. I would like to stay with ArchLinux if possible. It is so lovely, straightforward and lean/clean. Nothing what is not needed. And the community seems great, too! Thank you, too!

seth · 2026-07-09 20:28:29

You'll have to share the url you got

Making systemd units is not at all easy for me.

/etc/systemd/system/mynetwork.service

[Unit]
After=systemd-udevd.service
Description=Setup network
Before=network.target
Wants=network.target

[Service]
Type=oneshot
ExecStart=/usr/bin/myscript.sh
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

sudo systemctl enable mynetwork

https://ewontfix.com/15/

sighlinux · 2026-07-10 09:38:41

seth wrote:

You'll have to share the url you got

I got no URL, thought maybe it is your website so you see it anyway.
I just looked again and noticed that the domain is blocked by some blocklist the host does not respond when pinging. So curl probably has timeouted quietly.
So I posted it to termbin instead.

Today morning I spent several hours unsuccessfully trying to find out what causes the strange behaviour that apparently an old, overwritten script version gets executed, instead of the current one. (This is bizarre, I do not understand this. It makes me doubt of my mental sanity. Probably I just overlooked something, and need to sleep more.)
I think this must be resolved first, because my problems with systemd units is more getting them work than writing them
Whatever, later today I will try that sample you gave me... if it happens to work, I'll be happy, even if I don't understand why it works

Maybe you can spot the issue looking through the log?

Last edited by sighlinux (2026-07-10 11:07:00)

seth · 2026-07-10 12:33:21

There doesn't seem to be any enp* but eno1/2 are there after the root switch but before crond and anacron start (the latter actually only after yorur login)
The NICs actually only

Jul 10 06:35:08 archlinux kernel: Linux version 7.0.11-arch1-1 (linux@archlinux) (gcc (GCC) 16.1.1 20260430, GNU ld (GNU Binutils) 2.46.0) #1 SMP PREEMPT_DYNAMIC Tue, 02 Jun 2026 18:26:58 +0000
…
Jul 10 09:10:41 llamas kernel: igb 0000:02:00.0 eno1: igb: eno1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
Jul 10 09:10:41 llamas kernel: igb 0000:03:00.0 eno2: igb: eno2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

go up almost 3h after the boot??

sighlinux · 2026-07-10 15:56:20

seth wrote:

There doesn't seem to be any enp* ...

yes, these show up only as aliases with ip a.

...go up almost 3h after the boot??

yes, that was when I manually ran my network setup script to upload the boot log.

Anyway, I found the problem cause:

Archlinux apparently IGNORES /etc/crontab !!!

The ONLY correct way to edit the crontab on Archlinux seems via crontab -e.

When you try this, the system will throw an error that it cannot find /usr/bin/vi.

You might consider to set a symbolic link to vim. But this is a very bad idea.
Because: These horrible colors. The totally wrong edit/cursor movement mode handling.
And much more... for example, search/replace does not work in vim.
This all is super annoying if you are accustomed with vi for 35 years.

So the correct solution on Archlinux is pacman -Sy vi, and then crontab -e instead of vi /etc/crontab.

seth · 2026-07-10 17:18:07

Not sure about arch, but cronie is supposed to handle that file, https://man.archlinux.org/man/crontab.5.en#FILES

sighlinux · 2026-07-10 19:28:32

seth wrote:

...cronie is supposed to handle that file...

Yes, I took that for granted, and that made me not consider the possibility that for some unknown reason maybe /etc/crontab might even be ignored, until I ruled out other possibilities like overlay mount issues, bad inode/file mapping or differing namespaces. :facepalm
crontab -e puts the files into /var/spool/cron/<username> instead of editing /etc/crontab...

I guess it could be appropriate to file a bug report against cronie:
1. the apparent ignoring of /etc/crontab
2. what to do if vi is not encountered on the system (maybe running vim in this case, instead of throwing an error?)

I am just not yet sure whether to file on the Archlinux cronie package page or on the github cronie issues page.

Anyway, I am happy now The server boots up fine now ))

Many many thanks to you and Lone_Wolf

seth · 2026-07-10 20:39:51

https://man.archlinux.org/man/crontab.5.en#CAVEATS
Is /etc/crontab still root:root@644 ?

sighlinux · 2026-07-11 09:01:29

seth wrote:

https://man.archlinux.org/man/crontab.5.en#CAVEATS
Is /etc/crontab still root:root@644 ?

Yes, I created it with root:root@644 from beginning.

I made some checks again to find out what was the actual issue.
So it must have been that I maybe in a previous edit I did not terminate the one-liner file with LF but instead with EOF.
I consider this a POLA violation, as the Unix cron does not have this limitation. This limitation also is not supported by POSIX. I haven´t encountered that issue on any BSD. This is apparently a cronie-specific issue.

This makes clear that the correct place to file the bug report is the github cronie issues page.
Thank you again, @seth.

cryptearth · 2026-07-11 14:11:00

sorry for pouring in about a metric ton of salt - but may i ask: was all that worth it over just placing two rather simple files in a given folder and enable one service?

let's address several points:

aside from the two man pages you referenced in your last post being 25 years old - these are the wrong ones
Sun doesn't even specify a man5 for the crontab file format - and so doesn't posix neither
as for your "i haven't encountered this on bsd": well - if you look at the correct man pages: https://www.unix.com/man-page/linux/5/crontab/ and https://www.unix.com/man-page/netbsd/5/crontab/ (both from 2010) there's the obvious difference at the bottom:

cron requires that each entry in a crontab end in a newline character. If the last entry in a crontab is missing a newline (ie, terminated by EOF), cron will consider the crontab (at least partially) broken. A warning will be written to syslog.

it's right there in the specific man page for linux: on linux every crontab file requires a "\n" as terminator
so your conclusion that unix cron doesn'T have this limit is validated by your link - as you link in SunOS 9(!) from 2001 - that's not unix from the 70's(!) - and neither is your conclusion about this "limitation" (which it isn't) isn't support by posix - as posix doesn't even have any spec about the file format itself!

about using crontab -e over editing any file directly - and about using a different editor: this is explained in BOTH of your linked man pages:

The following options shall be supported:
-e Edit a copy of the invoking user's crontab entry, or create an empty entry to edit if the crontab entry does not exist. When editing
is complete, the entry shall be installed as the user's crontab entry.
// ...
The following environment variables shall affect the execution of crontab:
EDITOR Determine the editor to be invoked when the -e option is specified. The default editor shall be vi.

The following options are supported:
-e
Edits a copy of the current user's crontab file, or creates an empty file to edit if crontab does not exist. When editing is complete, the file is installed as the user's crontab file. If a username is given, the specified user's crontab file is edited, rather than the current user's crontab file; this may only be done by a user with the solaris.jobs.admin authorization. The environment variable EDITOR determines which editor is invoked with the -e option. The default editor is ed(1). Notice that all crontab jobs should be submitted using crontab; you should not add jobs by just editing the crontab file because cron will not be aware of changes made this way.

NOTE: while the 1p man page states vi as default editor Sun says it should be ed - but, if you don't like either of them, you can use nano:

EDITOR=nano crontab -e

filing a bug report should result in "invalid - user error" - you're on linux and have to follow the linux rules - you just can'T transfer bsd or even unix rules one-to-one

cron has to be run as systemd service anyway: https://wiki.archlinux.org/title/Cron

After installation, the daemon will not be enabled by default. The installed package likely provides a service, which can be controlled by systemctl. For example, cronie uses cronie.service.

so - you now have to used systemd anyway to start a service which is about to run a malformed config file with a somewhat questionable custom script instead of use systemd to use a built-in service managing your network directly
sorry - but aside from i not getting why you even want to go that extra mile in troduces potential additional errors - as you experienced by doing something different to how it's expected to be done

btw: how do you start and run llama? as systemd service? also via cron? this comes back around again: you try to enforce your bsd way which just doesn't work on linux - both are different - and linux isn't even a direct decendant but an acutal rewrite - comparing it to unix the same way as bsd to unix is already flawed

overall i recommend: https://wiki.archlinux.org/title/Network_configuration - and get used to: you are on arch now - you have to do it the arch(/linux) way - please try to let lose your (seemingly) outdated bsd/unix/posix hardwires in your brain - it will cause you a lot of issues otherwise

seth · 2026-07-11 14:40:53

https://aur.archlinux.org/packages?O=0&K=dcron
cron is not standardized at all, there're various somewhat compatible versions.

Arch Linux

#1 2026-07-08 20:05:15

[SOLVED] Setting up network at boot without systemd

#2 2026-07-08 20:22:30

Re: [SOLVED] Setting up network at boot without systemd

#3 2026-07-08 21:06:22

Re: [SOLVED] Setting up network at boot without systemd

#4 2026-07-08 21:10:37

Re: [SOLVED] Setting up network at boot without systemd

#5 2026-07-09 07:50:59

Re: [SOLVED] Setting up network at boot without systemd

#6 2026-07-09 09:45:55

Re: [SOLVED] Setting up network at boot without systemd

#7 2026-07-09 15:11:03

Re: [SOLVED] Setting up network at boot without systemd

#8 2026-07-09 19:19:01

Re: [SOLVED] Setting up network at boot without systemd

#9 2026-07-09 20:28:29

Re: [SOLVED] Setting up network at boot without systemd

#10 2026-07-10 09:38:41

Re: [SOLVED] Setting up network at boot without systemd

#11 2026-07-10 12:33:21

Re: [SOLVED] Setting up network at boot without systemd

#12 2026-07-10 15:56:20

Re: [SOLVED] Setting up network at boot without systemd

#13 2026-07-10 17:18:07

Re: [SOLVED] Setting up network at boot without systemd

#14 2026-07-10 19:28:32

Re: [SOLVED] Setting up network at boot without systemd

#15 2026-07-10 20:39:51

Re: [SOLVED] Setting up network at boot without systemd

#16 2026-07-11 09:01:29

Re: [SOLVED] Setting up network at boot without systemd

#17 2026-07-11 14:11:00

Re: [SOLVED] Setting up network at boot without systemd

#18 2026-07-11 14:40:53

Re: [SOLVED] Setting up network at boot without systemd

Board footer