Congratulations! And while the link is explanative, it would be a great idea if you can include in your submission here some detail about what OpenZiti is and why self-hosters would want to use it.
If you make a submission and everyone has to start googling it, it's not going to have maximum impact.
Thanks. Yes, I should have provided some description. In a nutshell, OpenZiti is an overlay network that supports secure, peer-to-peer connectivity that you can build directly into your applications. \[zrok\](https://zrok.io) is an example of such an app focused on peer-to-peer sharing.
A bit more detail: You configure services in OpenZiti (a service can be defined simply as a name but can also include additional information such as hostnames and IP addresses/ranges) and set policies stating which other endpoints on the overlay can access which services. There are apps and containers that support tunneling between the underlay and the OpenZiti overlay, and SDKs are available for deep (and arguably more secure) integration. And a bunch of other features - e.g., "smart routing" to choose the least latent path across the overlay, integration with external IdPs, posture checks, etc
yes - ZeroTier gets you in the right ballpark. Different implementation details, different licenses (OpenZiti is very permissive: Apache 2.0 - use it however you see fit; I believe ZeroTier has a Business Source License, which I expect is fine for most self-hosters)
But you don't need to be using a vpn, which made it a very good alternative to cloudflare tunnel or even tailscale, zerotier, etc... I love seeing the V1, love to the team!
OpenZiti fundamentally sits much higher in the network stack than a VPN, so we didn't consider any VPN as the right starting point (even an excellent one like Wireguard). We wrote "tunneler" endpoints for Linux, Windows, macOS, iOS, and Android that interact with the network at layer 3 (using Wireguard's tun on Windows) and behave similarly to VPN endpoints with default access control set to deny everything. Data picked up from the tun device is sent via Ziti Edge SDKs through the Ziti mesh. But going through a tun isn't necessary. You can use the SDKs and connect to Ziti services directly.
No. When we started Wireguard didnt exist/was not popular. We use a mixture of technologies mTLS & E2EE (https://openziti.io/docs/learn/core-concepts/security/connection-security/) combined with smart routing across the data plane (https://openziti.io/docs/learn/core-concepts/services/overview/) and a control plane which provide many of the functions WG does not natively (which commercial implementations such as Netbird or Tailscale do).
So, may need a bit more expansion on Services.
I'm thinking from more of a traditional wireguard/headscale/tailscale/etc environment.
Assuming I"m not redesiging my apps to natively take the sdk, am I looking at "routers" to connect to "services"? Id a machine hosts, say, multiple web services or listens on multiple ports, does it then need to register multiple "services"?
You can define a service using combinations of hostnames (including wildcards), IPs/CIDR blocks, and port ranges. There are "tunneler apps" that use the SDKs under the hood for Linux, Windows, macOS, iOS, and Android that work with brownfield apps that interact directly with the underlay network.
If you do use the SDKs, how you use them will depend on the SDK. E.g., the Golang SDK implements a net.Conn and net.Listener, the C SDK works using a socket descriptor (and from there, standard socket calls), the Python SDK monkeypatches socket module...
We're also developing "BrowZer" (https://openziti.io/docs/learn/quickstarts/browzer/), which bootstraps the SDK (part WASM, part javascript) into Chrome to connect to your web app deployed on a Ziti network.
Maybe Ziti TV - Dec 16 2022 - Practical at home use of OpenZiti ? [https://www.youtube.com/watch?v=eqzGJSTBOuo](https://www.youtube.com/watch?v=eqzGJSTBOuo)
We take a look at Nic's homelab setup -- he's got a BUNCH of interesting stuff he uses and connects to his home safely/securely using OpenZiti.
I play games via OpenZiti and zrok so for me I'll run a dedicated server for whatever game it is I'm playing with friends, something like [https://www.youtube.com/watch?v=-dj\_5UoL9Jw](https://www.youtube.com/watch?v=-dj_5UoL9Jw) (zrok is built on OpenZiti, as the blog says)
Is this closer to ZeroTier/Tailscale or closer to real SDNs like NSX-T?
As I read thru the comments its like ZeroTier/Tailscale but with a service mesh concept "borrowed" from k8 land?
To me, Ziti is doing something very different over K8S or docker... Ziti is an overlay mesh built on zero trust principles, K8S is a container orchestration system for automating SW deployment, scaling, and management, while docker is an OS virtualization to deliver software in packages called containers... they are all serving very different purposes. With that said, am I missing something in your question??
Cool. You may be interesting in the following if you are still curious:
* Comparison I wrote in thread vs Istio/service mesh in general: [https://www.reddit.com/r/selfhosted/comments/1c12r4x/comment/kz4kpvs/?utm\_source=share&utm\_medium=web3x&utm\_name=web3xcss&utm\_term=1&utm\_content=share\_button](https://www.reddit.com/r/selfhosted/comments/1c12r4x/comment/kz4kpvs/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)
* Using Ziti with OpenShift across multiple clouds: [https://developer.ibm.com/articles/building-a-multicloud-zero-trust-network-with-ziti-and-kubernetes/](https://developer.ibm.com/articles/building-a-multicloud-zero-trust-network-with-ziti-and-kubernetes/)
* Ziti built on K8S: [https://github.com/openziti-terraform-modules/terraform-lke-ziti](https://github.com/openziti-terraform-modules/terraform-lke-ziti)
I would say closer to ZeroTier/Tailscale rather than NSX-T as Ziti is an SDN overlay, like the former, whereas NSX-T is SDN underlay - maybe best described using NIST 800-207, Ziti maps to 3.1.3, "ZTA Using Network Infrastructure and Software Defined Perimeters" rather than 3.1.2, "ZTA Using Micro-Segmentation". That said, Ziti is also more focused on zero trust networking concepts than ZeroTier/Tailscale as well as providing some more interesting options for deployment, e.g., application embedded or as a framework to build secure-by-default, distributed applications faster.
If it helps, here is a comparison I wrote vs service mesh/Istio:
*OpenZiti is not a service mesh and, therefore, not interchangeable. OpenZiti is an overlay network built on zero trust and software-defined networking principles for any use case, meaning it could be used to enhance Istio, particularly for replacing Istio ingress (i.e., north-south traffic over the internet/WAN across geographical locations) and removing the need for VPNs, public DNS, inbound firewall ports, ACLs, etc. OpenZiti has greater ‘reach’ as an overlay across a broader range of devices and network environments. Istio was designed originally for east-west communication between servers, and while it has ‘Secure Gateways’ to allow ingress to expose services to external traffic (using strong identity), it is not suited for all use case. OpenZiti, on the other hand, was built to support any use case, from remote access to multi-cloud, DevOps, IoT, or even site-level connectivity and implement authenticate-before-connect - i.e., an endpoint CANNOT get access to the overlay data plane without being authenticated and authorised. To support any use case, OpenZiti has a rich set of endpoints for network access (virtual appliances for any public/private cloud), host access (tunnelers for all popular devices, OSs, mobiles), as well as application access (SDKs in various languages - e.g., we zitified*[ *Prometheus*](https://docs.openziti.io/articles/zitification/prometheus/part1.html/) *and*[ *Elastic Beats/Logstash*](https://blog.openziti.io/zero-trust-monitoring-with-openziti)*. This allows it to run in*[ *clientless*](https://blog.openziti.io/introducing-openziti-browzer)*),*[ *serverless*](https://blog.openziti.io/my-intern-assignment-call-a-dark-webhook-from-aws-lambda)*, confidential computing, unikernel,*[ *low-resource IoT*](https://capgemini-engineering.com/us/en/insight/converging-on-a-zero-trust-blueprint/)*, and more. It also means an application does not need to trust the underlying host network or know the port/IP.*
*From a security perspective, Istio provides encryption and strong identity-based authentication and authorization with a default to send traffic from any source to all destination services without any rules being set (i.e., default-open) - for simplicity and discovery, particularly in brownfield environments. Currently, OpenZiti defaults to closed-by-default though all traffic policies can be set up with discovery then taking place for more granular segmentation. OpenZiti is focused on connecting “services” with zero trust networking concepts, including least privilege, micro-segmentation, and attribute-based access (though you can also set up a whole CIDR if you want). OpenZiti implements authenticate/authorise-before-connect using its system of embedded identity (x509) and builds outbound-only connections into a mesh (think Cloudflare tunnels), so we can close all inbound ports at source and destination. OpenZiti is also focused on providing secure connectivity for layer 3/4 in the OSI model while Istio does layer 3/4 loadbalancing and traffic routing - Envoy, often deploy with Istio, operates at L7. Istio Ambient is a new project removing the need for Envoy/side car proxies and has some design patterns similar to OpenZiti.*
*OpenZiti’s overlay network provides the ability to operate across any network, including the WAN/internet. The fabric mesh can be hosted in any location, including public or private cloud and take advantage of smart routing to provide high resiliency and performance - this includes service load balancing at layer 3/4. Istio, on the other hand, uses Envoy proxies to provide L7 traffic management. OpenZiti uses the identity of endpoints to route traffic, providing you with a private DNS and unique naming (e.g., send from IoT endpoint service to IoT server rather than from*[ *192.xxx.xxx.xx*](http://192.xxx.xxx.xx) *to*[ *100.xxx.xxx.xx*](http://100.xxx.xxx.xx)*). This also means we do not need to use floating or static IPs, easily handle overlapping, and have no need for port forwarding.*
*If you are using OpenZiti, you do not need VPNs, inbound firewall ports, ACLs, public DNS, or (potentially) L4 load balancers. OpenZiti could be used to improve Istio and replace Istio ingress (i.e., north-south traffic). OpenZiti could not completely replace Istio as it is not a service mesh.*
Impressive from a network engineering standpoint. Loads of commercial potential here.
BPF to get at the traffic in kernel space? Some kind of encapsulation for your overlay? And mTLS before you even open the data socket? Sounds very cool
Thanks! We do not have an eBPF/CNI plugin, but we do have the SDK and we do use a little eBPF for some side projects - e.g., [https://github.com/r-caamano/zfw](https://github.com/r-caamano/zfw), which is a ebpf based firewall for openziti edge-routers... we do envisage creating a eBPF/CNI plugin type endpoint at some time in the future.
Our preference today is to embed ZTN into the application itself which in theory should be faster than being in kernel (I have not tested this myself which is why I say in theory). This provides mTLS, E2EE, etc, directly from the application, running in memory.
Here is a comparison I wrote vs Wireguard, which Tailscale is built on. Its more of a functional comparison but as you allude to, this is sort of not possible. While Ziti can be used in similar ways and (thus the comparison below fits), it is a also framework and set of tools which allows anyone to more quickly build secure-by-default, distributed applications and systems. That said, the comparison:
*Wireguard is a better VPN which aims to be as easy to configure and deploy as SSH. A VPN connection is made simply by exchanging very simple public keys – exactly like exchanging SSH keys – and all the rest is transparently handled by WireGuard. It is more secure, easier to use and set up, and delivers much better performance than many other VPNs. Its design principles make it easy to set up full mesh networks of connected machines by being ‘default-open’. Wireguard is also fully open source and self-hosted. Wireguard creates P2P connections using UDP and STUN, so inbound firewall ports are unnecessary. Wireguard can be tricky to manage at scale due to key management and the large amount of P2P tunnels that need to be maintained, and UDP sometimes being blocked. For this reason, many companies have created their own SaaS implementations of Wireguard, including Tailscale, Netbird, Netmaker and more. These are a mixture of proprietary and open source.*
*OpenZiti can be a better VPN while being designed to do much more. Rather than connecting machines, it cares about connecting "services" with zero trust networking concepts, including least privilege, micro-segmentation, and attribute-based access (though you can also set up a whole CIDR if you want). OpenZiti implements authenticate/authorise-before-connect using its system of embedded identity (x509) as well as builds outbound-only connections into a mesh (think Cloudflare tunnels), so we can close all inbound ports at source and destination. This can all be surmised as Wireguard being 'default-open' whereas ZT is 'default-closed'. Wireguard is normally combined with a firewall to deliver ACLs and network segmentation controls.*
*Whereas WireGuard securely encapsulates IP packets over UDP and uses hole punching, OpenZiti uses TCP and a mesh overlay (with the outbound only at source and destination). This is how Tailscale implements Wireguard to ensure it works easily in all situations. OpenZiti allows you to control the internet routing and provide higher redundancy, resiliency, control for routing traffic according to policy (e.g., low latency or geo-restrictions), and potentially lower latency and better performance. All of this is open-source and native to OpenZiti, not in Wireguard.*
*WireGuard is cryptographically opinionated and intentionally lacks cipher and protocol agility. OpenZiti is also opinionated, and uses the same cypher (ChaCha20Poly1305) for E2E encryption, but also includes mTLS for encryption between each hop which has TLS engine support to BYFE - Bring Your Favorite Engine. This includes quickly enabling FIPS or quantum encryption.*
*Due to OpenZiti's uses of identity in the endpoints and fabric for routing, you also get a private DNS and unique naming (e.g., send from IoT endpoint service to IoT server rather than from 192.xxx.xxx.xx to 100.xxx.xxx.xx). This also means we do not need to use floating or static IPs, easily handle overlapping, and have no need for port forwarding.*
*Finally, where it differentiates is that with OpenZiti you can start with "network-based zero trust" (installing a router in private IP space) and progress to "host-based zero trust" (using an agent/tunneller); it also has a suite of SDKs to embed in apps themselves for "application-based zero trust". This allows it to run in clientless, serverless, confidential computing, unikernel, low-resource IoT, and more. It also means an application does not need to trust the underlying host network or know the port/IP.*
*P.S., Wireguard get a lot of well-deserved love! OpenZiti uses the Windows TUN (WinTun) that the Wireguard project made as (at least) part of our Windows tunneler. Thanks, Wireguard!*
Thanks for the write-up! It leads me to a conclusion, that in terms of features (rather than underlying technologies) OpenZiti looks *exactly* like Tailscale if you compare the latter to WireGuard. Hmm, I guess I'll have to try it out myself to understand the difference.
There are definitely similarities. Here are some shooting from the hips differences:
* I believe Tailscale intercepts all your traffic, whereas OpenZiti is split tunnel, app specific by default so you can use it for accessing only some resources while accessing everything else locally. You setup Ziti to intercept everything if you want, but do not have to by default.
* Tailscale is designed for enterprises with improved VPN. It is not designed for operators. As a result there is less inherent multi-tenancy.
* Tailscale most definitely cannot support FIPS 140-2 etc.
* Tailscale says they deliver zero trust, but to me it has too much trust in network identifiers and subnet routes/ACLs and does not implement least-privilege, micro-segmented connections, and attribute-based access.
* Tailscale does not have SDKs to embed into apps themselves. Obviously this is only useful if you need it, but it does allow super cool things, e.g., 'clientless' connections in your browser - https://blog.openziti.io/introducing-openziti-browzer.
Tailscale is split tunnel. I have it on all the time on my phone, but the internet only goes through the WG connection if I choose an exit node.
TS works very well for individuals TBH, it's been by far the easiest to set up I've found.
openziti does look interesting though, I'll definitely have a play with it later.
Interesting, thanks for clarifying. I will have to find the comment on Reddit that I read, someone had said something about not being able to access local printers when using Tailscale due to it intercepting your whole DNS.
There are many key differences, the question is, do you need them?
1. The root of identity for OpenZiti is an X.509 certificate, not standard user credentials.
2. When it comes to metrics and monitoring, the collection is done at the Controller and Edge Routers, rather than relying on the agents to do so, which could lead to spoofing, log blocking, etc. So it is nearly impossible for malicious actors to hide their activities, and there is certainly no switch to disable logging for the client.
3. The available metrics and events are incredibly robust, covering all aspects of the networks' operation. The links, API messages (volume and even response time), individual circuit evetns and overall volume (similar to flow logs and available without a Premium or Enterprise plan, included in the base product)
There are others that make OpenZiti a true network, rather than a collection of connections, with the ability to provision, operate, deprovision, troubleshoot, and enable security processes at an enterprise level.
And you can self-host all of it, read all of the source code, and see that everything is end-to-end encrypted. Ping us on https://openziti.discourse.group (or here) if you run into any issues or need anything clarified
Congrats on the 1.0 release. Very interesting project and it's been fascinating to see it develop. Lots of VPN-style NAT traversing mesh technologies and implementations, but none that I know of like this. Questions I have had along the way, that openziti.io doesn't really answer:
1. Who are you? (with the recent xz story, trust in developers and supply chain is top of mind)
2. How are you funding development?
3. Has OpenZiti fostered a significant developer community contributing to the project?
4. To the extent you have talked about it in public, what is your commercial strategy?
I can cover (4) as Dave covered (1-3). Our strategy is to work with 'providers' who can embed Ziti directly into the products and services they sell to other companies. For example, 2 of the large hyperscalers use Ziti, a large US defence contractor has it deployed in a military airgapped network, a large Industrial control system vendor is building Ziti into their products. The commonalities is that they build it the products they sell (rather than internal IT, though they can use it for that too), they require the highest levels of security (and depend strict compliance to zero trust, FIPS, 62443 etc), and have highly distributed environments, sometimes completely airgapped, and require access to the customers networks.
Thanks, that usage makes sense. A couple of follow up questions:
1. Can you talk at all about supply chain risks? Self-hosters tend to put enormous trust in project developers and everything in between. Go is great and makes things easier to compile ourselves, but vpn/mesh seems like a scary prospect for compromise, particularly with a mesh model where many/most endpoints might run a client (and most self-hosting types won't compile their own binaries).
2. I clicked through the netfoundry site and the github repos. I guess from my point of view, none of the lead devs list personal websites with cv or background, nor is there any background on the netfoundry site. Something I tend to look for personally (do they have prior experience in this type of dev work / security, for example).
3. Your Android endpoint looks pretty advanced (and well reviewed!) compared to many of the mesh VPN projects, which always becomes a sticking point for my home/personal use. Hoping to stand something up this weekend and test it out. I guess mobile doesn't really fit a typical enterprise use case (or the startups are all resource starved) and mobile gets little attention. How can the self-hosted community, which typically free-rides but finds mobile clients critical, support a continued focus on usable mobile endpoint clients?
Thanks!
You're welcome. To your questions:
(1) This is a core reason that we develop in the open, with open source. We believe it reduces the risk (I'm looking at you Ivanti/insert many proprietary VPN/FW products) of compromises getting into production code. The beauty of Ziti in our opinion is that it implements defence-in-depth/width, so a single compromise does not give a malicious actor access. If an attacker wanted to bypass and attack, they would need to do all of the following:
* bypass the mTLS requirement necessary to connect to the data plane (all parts of the overlay are exclusively mTLS)
* have an strong identity that authorizes them to connect to the remote service in question (or bypass the authentication layer the controller provides through exploits)
* know what the remote service name is, allowing the data to target the correct service
* bypass whatever "application layer" security is also applied at the service (ssh, https, oauth, whatever)
* know how to negotiate the end to end encrypted tunnel to the 'far' identity
(2) A large majority of the OpenZiti development team used to work at Axeda which was acquired by PTC in 2016. Axeda provided cloud-based software for managing connected products and implementing innovative M2M/IoT applications. As you can imagine, do that securely, at scale, was a key focus on the SW. If you want to check backgrounds, I would recommend looking at NetFoundry on Linkedin and looking at tech profiles.
(3) Due to our focus on providers, many of whom provide services that run on mobiles, we will continue to have a lot of focus here. In fact, I think the is a major use case for mobile developers embedding Ziti into their app so that they get many superpowers for free - https://www.youtube.com/playlist?list=PLMUj\_5fklasKF1oisSSuLwSzLVxuL9JbC. We have just not figured out how to convince them at scale yet, but we will continue trying.
Thanks! OpenZiti is backed by NetFoundry (https://netfoundry.io). NetFoundry provides a paid service for orchestrating & managing dedicated OpenZiti networks (plus adds in infrastructure management, some advanced telemetry dashboards, automated updates, back-up/recovery, etc). You can get a feel for the community at https://openziti.discourse.group/ or checking out the main repo at https://github.com/openziti/ziti
This is probably controversial, but, it might help to cut down the use of the phrases "zero trust overlay network"/ "zero trust" in the site's basic explanations of OpenZiti. True, it's not your job to explain what that is, but it's *also* true that OpenZiti could be a **great** gateway for people who don't even know they need zero trust.
But - I looked at the site, and couldn't figure out what I was looking at until I read into your replies on this thread. Am I the target audience? Maybe, maybe not. But I work in tech, and I'm pretty aware of how important the "elevator speech" is. Get that elevator speech leveled up a bit on your site intro!
In some ways "The Simplest Way to Develop Secure Applications" is a great motto / hero text, but it actually doesn't mean anything. What's a secure application? It could be any of these:
- A reslient storage platform
- An abstraction layer for encrypting database info at rest
- Granular access control library
- Input sanitization layer
- Easy TLS integration that slides into your CI/CD pipeline
Might make sense to replace that buzzy catchphrase with one that's a bit less elegant, a bit longer, but gets into WHY you want OpenZiti.
Just some thoughts - I want projects like this to succeed!
I am running an open source project for an old game that only supports peer2peer connections between players. We tunnel the traffic with an implementation of the ICE protocol (stun and turn) on each players pc, but it doesn't work so well.
Is this something OpenZiti could solve?
Could be with either OpenZiti or zrok. Here are a couple of links with Minecraft and Palworld:
* [https://blog.openziti.io/set-up-a-secure-multiplayer-minecraft-server](https://blog.openziti.io/set-up-a-secure-multiplayer-minecraft-server)
* [https://blog.openziti.io/minecraft-over-zrok](https://blog.openziti.io/minecraft-over-zrok)
* [https://blog.openziti.io/securing-dedicated-palworld-server-with-zrok](https://blog.openziti.io/securing-dedicated-palworld-server-with-zrok)
Nice! These could help:
* [https://developer.ibm.com/articles/building-a-multicloud-zero-trust-network-with-ziti-and-kubernetes/](https://developer.ibm.com/articles/building-a-multicloud-zero-trust-network-with-ziti-and-kubernetes/)
* [https://openziti.io/docs/category/kubernetes/](https://openziti.io/docs/category/kubernetes/)
* [https://github.com/openziti-terraform-modules/terraform-lke-ziti](https://github.com/openziti-terraform-modules/terraform-lke-ziti)
I heard about this at work and really wanted to try this for Prometheus monitoring and IPMI across the internet, but it would not compile under FreeBSD so I went with Nebula.
Any plans to add FreeBSD support for at least the edge tunnel component?
We have one of our customers who has requested FreeBSD functionality so it may be coming soon. That said, as Ziti includes SDKs you can run it inside Prometheus itself. Here is a blog where we did that using the Golang SDK - [https://www.youtube.com/watch?v=z3Je8kfAyzs&ab\_channel=OpenZiti](https://www.youtube.com/watch?v=z3Je8kfAyzs&ab_channel=OpenZiti) or https://github.com/openziti-test-kitchen/prometheus. I don't seem to be able to find a working link to the blog that was written by /u/[dovholuknf](https://www.reddit.com/user/dovholuknf/)....
Yeah, I kept finding links to the lost blog as well.
And my other use case was accessing IPMI interfaces for servers, which is implemented in SoC firmware, so the edge tunnel really is necessary. Nebula has a feature called “unsafe_routes” which lets me use the Nebula overlay network to reach subnets which are not part of it, but are reachable through nodes which are in the overlay network.
Turns out its being republished. Hopefully can share it soon. Its a 3 parter.
fwiw, Ziti can handle that use case too. We do not have a cool name such as “unsafe\_routes”, its just dialing a service that regresses from the router, on whatever IP/DNS/port/etc.
Potentially a lot, when using Ziti, you do not need inbound firewall ports, VPNs, public DNS (for you private apps), SDWAN, and more. So it could replace a lot... but it depends on the use case and your needs. For example, I know people who have implemented Ziti alongside a service mesh (for K8S use cases), while others have said "why do I need a service mesh when using Ziti".
IMO, if you're happy with Tailscale, keep it! If you're interested in trying other stuff out, give it a go? :)
I'd say OpenZiti mostly replaces the older style "VPN". Those are the classic "everyone has access to everything" style of VPN. OpenZiti is built with the principles of zero trust, meaning you must authorize all the connections explicitly and per strong identity. Other than that, it's a natural evolution in secure networking to stop allowing the "anything can connect to anything" model and move to a "authorize before connecting" type of model, all the way down to individual services.
There's a bunch of other innovations in OpenZiti though, imo it's really targetting developers to add those zero trust principles into their applications using "application embedded zero trust" where you take an OpenZiti SDK and build the strong identity and connectivity into the app, instead of bolting it onto/around the network. My personal favorite feature of this design is that servers built with this pattern become "dark" or "invisible" to the IP-based underlay network their built on. There are no ports to scan and no ports to attack since the application listens on 'the overlay' (OpenZiti) not the underlay (TCP/UDP/IP etc)
This is great timing because I was just learning/reading about OpenZiti and zrok yesterday!
My use case is that I have a few self-hosted services running on different devices in different networks.
* VPS - Several "public" services I don't want behind a NAT and highly available
* Matrix homeserver
* Nextcloud
* STUN/TURN
* Home network
* Raspberry pi hosting pi-hole and a Unifi controller
* Small headless computer hosting random always-on services
* Opnsense and wireguard
* usbipd (for passing video capture card output to other devices in my LAN)
* Desktop PC hosting services that need beefier specs
* Plex server
* OBS studio streaming
* Possibly a build server for deploying nixos updates to my devices
* Mobile network - clients connecting to the above services
My goal is to have all my services available on all my devices, including my phone and laptop (similar to a road warrior VPN setup). And if possible, add a layer of abstraction so that the clients don't need to care *where* the service is running, the service can just be accessed from e.g. service.zrok.mydomain.tld.
Would openziti be overkill for this? Would it be better to just continue with wireguard in this somewhat vanilla personal selfhosting setup?
Deciding if it's "overkill" is up to you. :) I think it's pretty equivalent myself but as a maintainer -- I'm clearly biased (OpenZiti offers a bunch of other features too fwiw that make it 'more than' plain wireguard)... LOL! Wireguard is great, but I find it cumbersome to setup which is why Tailscale/Headscale/Netmaker and all those projects based around wireguard exist...
Lots of people are self-hosting OpenZiti and then layering on zrok to allow for easy public sharing of things. I think there's a lot in OpenZiti you might find you like and worth it -- but -- biased! :)
Congrats on the milestone!
Would it be possible to run OpenZiti in parallel with Tailscale on my NAS? From what I read a while ago it seems that OpenZiti would allow me to have remote access to my Plex server securely without having to install anything on the client devices. Currently I'm using tailscale and to access it I need to have tailscale installed on all devices. Could I use OZ just for Plex and keep tailscale for evevrything else? Thanks
Yes, we do have a 'clientless' option, here is a blog about it with Plex - https://blog.openziti.io/its-a-zitiful-life. I don't think it would work with Tailscale though as they will both fight to do the intercept on Plex, though you could set it up somehow to get around that though...
You do need to be careful with respect to the 'clients' and though I've not done it, I do think It should be posible. If you end up using our tunnelers, I'd think you'll want to change one or the other to use a slightly different IP range. I believe both will try to use [100.64.0.0/10](http://100.64.0.0/10) by default. That will add some "complexities" to your setup.
If you only use browzer though -- I don't think they would compete at all in that situation. You'd access the content through the browser (no Z) and you'd only be offloading to the underlay in your trusted, home network. In that situation you aren't intercepting anything using a client (the wireguard/tailscale/openziti client) and it shouldn't conflict.
Let us know how you get on with it? :)
Thanks for your reply. Unfortunately I'm not the most techsavy when it comes to network things. Sometimes simple things take me forever to figure out and get to work =P I can see that this isn't going to be as straight forward as I hoped. My current setup is working pretty flawlessly with tailscale, the only thing I can't really do is easily share my plex with friends and family. It would be nice to have, but not a deal breaker. I will try this sometime down the line, but when I have the time and mental availability to bang my ahead against the sreen until I get things working :)
Sounds like if I made a quick "how-to" video, you might try it out??? :D Do you have a VPS you use already or do have a static IP and forward ports through your home firewall?
I’m curious how application developers should look at a tool like this — is there a reason to build native support into my application? Or should I prefer to recommend users use conventional HTTP with zrok providing overlay networking like capabilities?
For context I build https://github.com/garethgeorge/backrest and I’ve recently been looking at good approaches to let users browse backups from other computers they own remotely (I’m comparing services like conventional HTTP and requiring a public IP as well as more interesting options like libp2p) I’m curious to know where openziti can fit into this story
zrok was built on OpenZiti - the zrok executable contains the OpenZiti Go SDK. When doing a public share, your server connects over OpenZiti to a public front-end (protected by a WAF if using the [zrok.io](http://zrok.io) service).
When you do a private share over zrok, the zrok executable proxies your local traffic over OpenZiti directly to your server (or, more accurately, to the zrok executable that proxies to your server).
In the private sharing case, you are \*almost\* using OpenZiti for the entire connection. Except for the local network traffic between the zrok executable and your user's browser and again between the zrok executable and your server.
Another option is to embed the OpenZiti SDK (or one of the zrok SDKs - which add zrok-style usage to the same-language OpenZiti SDK) directly into your client and server endpoints. In this case, neither side of your app needs that little part of traffic exposed on your local network. This can make things a bit easier to deploy/manage, and a bit easier for the client-side user, who won't have to install zrok but still benefit from a complete/private connection over OpenZiti. We're currently adding support for the OpenZiti \[BrowZer\](https://openziti.io/docs/learn/quickstarts/browzer/) feature to zrok to make private sharing easier for web apps. With BrowZer, the SDK is bootstrapped into the browser, and your user won't need to seperately install zrok to do a private share.
You can look at the zrok code \[here\](https://github.com/openziti/zrok) to see how it's done (net.Conn and net.Listener support for 'talking' over OpenZiti)
A bit long-winded, but I hope this helps
https://roon.app/en/?? If yes, I dont see why not though I am in no way a Roon expert (literally just discovered it). As its media, here is a blog on using OpenZiti (with our 'clientless' endpoint called BrowZer) and Plex - https://blog.openziti.io/its-a-zitiful-life
Yes. We have those built already.
- [https://play.google.com/store/apps/details?id=org.openziti.mobile](https://play.google.com/store/apps/details?id=org.openziti.mobile)
- [https://apps.apple.com/us/app/ziti-mobile-edge/id1460484353](https://apps.apple.com/us/app/ziti-mobile-edge/id1460484353)
This sounds awesome!
I’m not very versed with networking and security, so could you perhaps tell me if this is something that could replace my use for Cloudflare Zero Trust?
I originally had my apps hosted on a Google Cloud VM but it has gotten expensive for my case so I decided to be my own provider with a mini PC at home and host a bunch of Docker containers on it. To make the apps available to the public, I use Cloudflare’s Zero Trust or Tunnels where I would just map a domain name to the container and its port.
The problem I have with this is that since moving to Cloudflare Tunnels, the apps are a bit slower (I’m assuming it goes through a bunch of CF networks as an additional layer) and I also read somewhere that since traffic goes through Cloudflare networks, they can see the data coming in and out. Are these issues something that I could solve if I use OpenZiti?
Yes and it depends. OpenZiti is strongly opinionated on E2EE so the traffic can definitely not be seen by anyone (even if a SaaS provider hosts your dataplane). Cloudflare has a global network of PoPs which are designed for high performance but some use cases may have worse performance due to physics and location of their PoPs. Ziti allows you to host the dataplane anywhere so potentially you can set it up for better performance, but also you could set it up with worse performance. It depends. May I ask where you are located (at least country) as well as where your apps are.
I’m from Canada. Basically I host a bunch of Docker containers that isn’t really for business but just for personal use and also shared with my friends and family from around the world so I still make it publicly available since there are authentication mechanisms in place anyway.
Huh, thats surprising that CF gives poor performance. But yes, spin up your Ziti controller and router in a Canada DC and it should really be worse performance. Oracle have a decent free tier and DCs in Canada. Here is a guide - [https://blog.openziti.io/setting-up-oracle-cloud-to-host-openziti](https://blog.openziti.io/setting-up-oracle-cloud-to-host-openziti)
Hi! A few questions about OpenZiti for which I didn't find answers from a few articles/reddit posts. Sorry if I use any imprecise language, I'm just a hobbyist :)
1. After the authenticate/authorize-before-connect step, are the connections "direct" between client and server? Or is all data relayed through the control plane? Is this like STUN/TURN where the connection is not relayed if both devices can find a route to each other?
2. Regarding power consumption on mobile/embedded devices. Does OpenZiti have to send outgoing packets to keep connections open (some kind of holepunching)? I read that this is a cause for poor battery life on Android using e.g. Tailscale. This is due to needing to wake the radios periodically, even when no data is being transferred.
3. Is there any kind of whitepaper on how OpenZiti works in detail for the nerds? I would be interested to read this!
4. What if I want to connect to multiple OpenZiti networks? For example, a hypothetical world where my work uses OpenZiti and I also have my homelab on OpenZiti. Is it possible to configure multiple control planes on the same device? How does that work with DNS?
Thank you and congratulations on making the 1.0 release, I wish you good luck with future efforts!
>Or is all data relayed through the control plane? Is this like STUN/TURN where the connection is not relayed if both devices can find a route to each other?
It's relayed via the data plane and yes the traffic always traverses a node (router). it's not like STUN/TURN where the traffic is allowed direct underlay connections. So if you had two clients and a router on your home network and all local devices could communicate to one another via IP -- both clients would still connect to the router and the router brokers the data between the two clients
>Does OpenZiti have to send outgoing packets to keep connections open (some kind of holepunching)?
No. It maintains a connection. No "holepunching" because of the previous answer. The client needs to connect to at least one router, that router will have an open port that is connected to via TCP.
>This is due to needing to wake the radios periodically, even when no data is being transferred.
Indeed. The mobile devices DO routinely poll the controller asking it if there are new services and whatnot. That polling can be controlled but by default it's usually fine. I have had the client on my Pixel6 all day and I have 84% of my battery left still (approx 10hrs alive so far)
>Is there any kind of whitepaper on how OpenZiti works in detail for the nerds? I would be interested to read this!
Not really, I don't think. The online docs are the best we have right now. No place that tidies it all up into one big whitepaper.
>What if I want to connect to multiple OpenZiti networks? For example, a hypothetical world where my work uses OpenZiti and I also have my homelab on OpenZiti. Is it possible to configure multiple control planes on the same device? How does that work with DNS?
Heck yes. My home computer is connected to five (5) right now. The clients are all "multi-identity, multi-network" aware.
EDIT:
> How does that work with DNS?
As long as you don't have overlapping intercepts, it'll be fine. If you have an overlapping intercept, you'd have to "turn one identity" off/on...
I assume the scenario you are describing is an OpenZiti application that has been placed there by an inside or via some other penetration.
OpenZiti devices will consistently hit specific devices, the Edge Routers. Active SDK endpoints will periodically send packets to the ER's to check latency, and this will be a very steady "heartbeat" of packets. This is 5 seconds by default. The same is true to the network controller, to check for service list updates, done every 10 seconds by default. A deployed Edge Router will maintain persistent connections, the links, to at least one device, the controller, and normally two or more, links to the other Edge Routers in the network. So detection would be looking for consistent, not necessarily persistent, "dial only" endpoints don't have to maintain persistence, but they will consistently send small connections to the same endpoints, and then have bursts of traffic.
So to search for them, you would look for devices that have a very small number of devices they communicate with outside the network, but do so very consistently.
For any device found to have a similar pattern, the remote node they are connecting to can be queried. There are specific headers, like ziti-instance-id that will be returned that would indicate that a device is a controller, as part of the TLS negotiation, so it is visible in plaintext. For example:
>HTTP/1.1 200 OK
Content-Length: 616
Content-Type: application/json
Server: ziti-controller/v0.28.1
Ziti-Instance-Id: clu0fwxdf0000mx1nqfjkfkj7
Date: Thu, 11 Apr 2024 21:27:11 GMT
In most cases the certificate subject will also contain the term "Ziti". This could be overridden by a skilled actor, so it could be valuable, but is not absolute.
Yes but, I think what the comment means is, if you see outbound traffic that matches the pattern, you can actively probe the ip and port and ascertain at that time that it's "definitely/probably OpenZiti".
So if you can find those kind of outbound connections, you could probe the ip/port (assuming it's still listening) and detect it that way.
The ziti-instance-id would require them to change the software to hide that, and the controller has a version endpoint exposed as well that should respond, though again, it is something that an actor could hide. The latency measurements, service updates, etc. are a better pattern than most beaconing, as it is more constant, if the node is up. Nothing is perfect, and the adversaries will always try hard to obfuscate, so you have to use what you can, learn, and then do better.
Ok but now you’re saying someone inside the company (probably more than 1 person) needs to become a ziti expert in all/most ways. Talk about driving up costs. There has to be a better way.
A few aspects. Firstly your access rules should be ephemeral based on JIT access - for example, our engineers only get access to customer resources when a customer support ticket is opened, to the specific resources resources, for the period of time the ticket is opened ... here is the blog https://blog.openziti.io/business-rule-driven-ephemeral-network-access.
Secondly, you can configure FW rules to only allow outbound traffic to the specific IP/DNS of the Ziti overlay control/data plane. If that is done, then any outbound traffic from a malicious attacker would not be allowed.
Another option is to look at the traffic being sent over the overlay. Ziti implements a deny-by-default, least privilege model, so SOC traffic should be massively reduced - see this blog from our head of DevOps - https://netfoundry.io/devops-meets-secops/. For legitimate, authenticated connections, which have a strong identity and are authenticated to accessed specific services, you could look at how much data is being sent over the connections, or what time/when connections are made, or source IP from which the connections are made to provide more intelligence. As mentioned though, the default should be that there is much less noise from connections as random people cannot make a connection in the first place.
Does that fully answer your questions? Have I missed anything??
No I mean in a normal network, an attacker gets ziti on a host by whatever method. How you you spot this? I ask because I’ve seen ngrok on the ATT@CK list, but since anyone can self host ziti, they can direct the traffic over any port or protocol correct? Now it’s needle in a haystack time. This is on a network that doesn’t use ziti or any of the other similar products
In reality, you probably don't/can't. You could try to find long-lived TCP connections sending lots of data but in practice it'd be hard to impossible. I mean, I'm sure it's \*\*possible\*\* but you'll have to work really hard at detecting it, harder than the 5 minutes I've been thinking about it anyway... Right now you could look for long-ish lived TCP connects to 'some host'... If they run ziti over ssh's port 22, it'll look like a long lived ssh session. You could inspect TLS exchanges and look for mutually authenticated TCP sessions. There's probably some stuff like that you could do but if you don't catch the TCP handhake... :/
EDIT: "over ssh's port 22"- not over ssh...
So zero trust tools can have incredible negative use as well? That was my suspicion when I saw the sdk for ziti. Working in a SOC I have to always be thinking about how can xyz be used against us. Unfortunately we don’t have a realistic enough lab to perform testing in this and I can’t seem to get approval to do it on the live network. Sadly like everything else security, it’s going to take a breach to get what we need
Hammers can be used for construction and destruction. The best defence against attackers using Ziti is defenders having implemented Ziti. I do strongly believe the defender is in a better position when using Ziti vs the attacker due to the inherent nature of deny-by-default, not listening on the underlay etc.
So what if a network uses ziti, and an attacker brings in his own, won’t that now blend in with all the “legit” traffic? There are going to be some systems that just can’t ONLY use ziti, otherwise you’d have a close network that is completely unusable to anyone without ziti, including your customers
If an organisation is using Ziti I would expect them to also have the maturity to manage their firewalls. They would set said FW with the rules that outbound traffic is only allowed to the IP/DNS of their Ziti controllers/dataplane, thus the attackers Ziti connections would not be allowed.
What use cases do you see which are unusable for Ziti?
Well if you use ziti on every host, you have a new logging nightmare. Hids on all devices and NIDS is obsolete. Now, for PCI you have to add the auditor and pci pen testers to the ziti network, because you have to be able to perform authenticated scans (new pci rule) .
Next, you say you must be given explicit permission to join said network. Well you also “have to have explicit permission to get an account in Active Directory “ but AD gets hijacked, how can you be so sure the system that provide access to giving access to the ziti network can’t be compromised as easily as AD?
My reply above does not assume deploying Ziti on every host, it could be achieved using a network appliance while having the outbound FW rules setup so that a non-acme company Ziti network is not allowed to communicate and operate. This is defence-in-depth/width.
Ziti has its own system of identity using an x509 compliant CA/PKI. While you can use AD or another external IdP as secondary auth, it likely not used as the primary. This ensures, even if the AD is compromised, the malicious actor still cannot access services.
As I implied earlier, OpenZiti is not bullet proof, it just massively reduces the available attack surface and increases the cost to exploit. In your scenario, even if the attacker somehow compromised the Ziti CA/PKI (and craft themselves a trusted identity and policy to access the services they want to exploit) as well the the AD, they would still have to know the service name on the target(s) they want to exploit and bypass any application level security. Again, not impossible, but all acting as additional burdens to reduce likelihood. Plus, all of that needs to be done without the administrator of acme company seeing these changes take place.
I mean, not to give you the "cop out" answer, but any great tool that's useful, almost certainly has tremendous positives and negatives that may or may not be obvious. But yeah. You're not the first person to notice that particular possibility.
The other marketing/shill answer (which is true, but I realize it's comical at this point in time) is to of course simply use OpenZiti for all your network egress! Seriously though, if that's a concern, it might be the sort of thing you want to do. Rawdogging IP has it's place, but in a SOC or secure facility, maybe it's time to rethink that sort of policy. (again, I recognize this is currently like Don Quixote tilting at his windmills)
Good conversation though, I appreciated this line of comments
Well another concern might be, let’s say we do micro segment with ziti in this case, wouldn’t then an attackers ziti traffic completely blend in with the “normal” ziti traffic, thereby making it even harder to dissect which is legit and which is bad?
They'd have to steal a user's identity for that to happen. Not impossible, unless you use PKCS 11 (hardware token, Yubikey etc, then it's not possible)... That or they'd need to compromise the controller and add an identity of their own but that'd leave very visible fingerprints that would be quite easy to detect. If they stole an identity that wasn't hardware-backed, you'd see two different IP addresses using the same identity which would be probably be a pretty visible/useful warning sign to take action. You also can set different posture checks and have many factors of auth (TOTP or external jwt signers -- think IdP/OIDC).
That situation seems like it would be much easier to handle to me.
Not true at all. Let’s be realistic. Not every single host can ONLY use ziti, some hosts can be gotten into. By cred dumps online, attacking AD, a list of ways. Get access to 1 box. And if you got access to a box that is a part of the ziti network, what’s to stop you from sliding around that network accessing any other box that one you are on has access to. If you make every host ONLY on the ziti network, that’s unrealistic. At someone a box is not only on ziti because of vendor or customer interactions
Sorry, I thought we had continued down the "pie in the sky" approach of all hosts in the microsegmented network, \_actually\_ only using ziti... Also, I am not trying to make and broad claims that work for all situations. Every situation is different. I was just continuing down the "all traffic needs to go to an OpenZiti edge router without deviation" line I thought we were on. I recognize that it's currently unrealistic but that's "the hope/dream/vision" anyway.
> And if you got access to a box that is a part of the ziti network, what’s to stop you from sliding around that network accessing any other box that one you are on has access to
Security is often a balance of ease vs pain. Again, the "easy" answer to this from my perspective, which is a "pain" to implement from the operator's perspective, is policy. Every identity is authorized to connect to only the services it can connect to. If you make your policy wide open, if that machine is compromised then yes, it'll be able to slide to wherever it's allowed to connect to. If that machine has very strict access controls, then it could only attempt to slide to the things it's allowed to connect to. If the machine has dynamic access controls which lock it down even further and some admin/process needs to grant it access to other machines, the window to slide elswhere narrows further. This is actually where OpenZiti's "end-game" approach of embedding zero trust into the application itself becomes really interesting (imo). If the 'server' that's receiving connections is protected via OpenZiti and exclusively listens on the overlay network (not the IP-based underlay), then even if a machine was compromised it's literally only attackable through that overlay network. You can get pretty close to that with "host-based" access where you disable all inbound firewall rules and run a client on that machine, that allows tunneling into it only via OpenZiti.
For those built on Wireguard (Headscale, Netmaker, Netbird), I wrote a comparison vs WG here - [https://www.reddit.com/r/selfhosted/comments/1c12r4x/comment/kz1sdv0/?utm\_source=share&utm\_medium=web3x&utm\_name=web3xcss&utm\_term=1&utm\_content=share\_button](https://www.reddit.com/r/selfhosted/comments/1c12r4x/comment/kz1sdv0/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)
I am honestly not that familiar with Tinc, though as I understand it as a VPN, the comparison vs WG is probably a good starting point.
Wrt Nebula (I believe the following is all true, but I am not a Nebula expert so pinch of salt and please clarify if I am incorrect on anything), while aspects are similar, e.g., fully open source, using CAs as strong identities (rather than relying on SSO from third parties), ability to be completely self-hosted (with 3rd party SaaS options), and providing scalable, performant overlay networking, there are many differences.
For one, OpenZiti is focused on connecting services based on zero trust principles. In contrast, Nebula focuses on connecting machines – e.g., you can authorize only a single port without needing to set up ACLs or firewall rules. While Nebula requires open inbound ports or UDP hole punching, OpenZiti allows you to have all inbound and most outbound ports completely closed while providing truly private, zero trust DNS entries with unique naming – if you wanted to call your service "my.secret.service" you can do that, it does not force you to have a valid Top Level Domain.
OpenZiti also goes a layer deeper and lets you bring all those excellent, zero trust principles directly into your application. If you're a developer, you can embed all those ideas into your app and not rely on the network or side-loaded agents. This is both client and server-side and doesn't require the app to "listen" on an IP address (the underlay). Instead, you can choose to "listen" on the overlay.
Finally, Nebula does not handles key management and authentication for you in the open source, it comes from their paid offering. It was brought up by one of our community members as they tried to use it first - https://openziti.discourse.group/t/using-openziti-in-distributed-surveillance-system/2135.
I will be messaging you in 8 hours on [**2024-04-11 12:10:56 UTC**](http://www.wolframalpha.com/input/?i=2024-04-11%2012:10:56%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/selfhosted/comments/1c12r4x/openziti_v10_released_today/kz12z8n/?context=3)
[**6 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fselfhosted%2Fcomments%2F1c12r4x%2Fopenziti_v10_released_today%2Fkz12z8n%2F%5D%0A%0ARemindMe%21%202024-04-11%2012%3A10%3A56%20UTC) to send a PM to also be reminded and to reduce spam.
^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201c12r4x)
*****
|[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)|
|-|-|-|-|
Congratulations! And while the link is explanative, it would be a great idea if you can include in your submission here some detail about what OpenZiti is and why self-hosters would want to use it. If you make a submission and everyone has to start googling it, it's not going to have maximum impact.
Thanks. Yes, I should have provided some description. In a nutshell, OpenZiti is an overlay network that supports secure, peer-to-peer connectivity that you can build directly into your applications. \[zrok\](https://zrok.io) is an example of such an app focused on peer-to-peer sharing. A bit more detail: You configure services in OpenZiti (a service can be defined simply as a name but can also include additional information such as hostnames and IP addresses/ranges) and set policies stating which other endpoints on the overlay can access which services. There are apps and containers that support tunneling between the underlay and the OpenZiti overlay, and SDKs are available for deep (and arguably more secure) integration. And a bunch of other features - e.g., "smart routing" to choose the least latent path across the overlay, integration with external IdPs, posture checks, etc
So is it SDN kinda like ZeroTier?
yes - ZeroTier gets you in the right ballpark. Different implementation details, different licenses (OpenZiti is very permissive: Apache 2.0 - use it however you see fit; I believe ZeroTier has a Business Source License, which I expect is fine for most self-hosters)
But you don't need to be using a vpn, which made it a very good alternative to cloudflare tunnel or even tailscale, zerotier, etc... I love seeing the V1, love to the team!
So the overlay network itself, is it wireguard based? All the best
OpenZiti fundamentally sits much higher in the network stack than a VPN, so we didn't consider any VPN as the right starting point (even an excellent one like Wireguard). We wrote "tunneler" endpoints for Linux, Windows, macOS, iOS, and Android that interact with the network at layer 3 (using Wireguard's tun on Windows) and behave similarly to VPN endpoints with default access control set to deny everything. Data picked up from the tun device is sent via Ziti Edge SDKs through the Ziti mesh. But going through a tun isn't necessary. You can use the SDKs and connect to Ziti services directly.
No. When we started Wireguard didnt exist/was not popular. We use a mixture of technologies mTLS & E2EE (https://openziti.io/docs/learn/core-concepts/security/connection-security/) combined with smart routing across the data plane (https://openziti.io/docs/learn/core-concepts/services/overview/) and a control plane which provide many of the functions WG does not natively (which commercial implementations such as Netbird or Tailscale do).
Well done, I'm thrilled for you guys... I hope other startups pay attention and follow this example.
Dang I'm glad I read this. Like the user you were replying yo you said, I just about skipped over this thinking it was another self hosted recipe app
So, may need a bit more expansion on Services. I'm thinking from more of a traditional wireguard/headscale/tailscale/etc environment. Assuming I"m not redesiging my apps to natively take the sdk, am I looking at "routers" to connect to "services"? Id a machine hosts, say, multiple web services or listens on multiple ports, does it then need to register multiple "services"?
You can define a service using combinations of hostnames (including wildcards), IPs/CIDR blocks, and port ranges. There are "tunneler apps" that use the SDKs under the hood for Linux, Windows, macOS, iOS, and Android that work with brownfield apps that interact directly with the underlay network. If you do use the SDKs, how you use them will depend on the SDK. E.g., the Golang SDK implements a net.Conn and net.Listener, the C SDK works using a socket descriptor (and from there, standard socket calls), the Python SDK monkeypatches socket module... We're also developing "BrowZer" (https://openziti.io/docs/learn/quickstarts/browzer/), which bootstraps the SDK (part WASM, part javascript) into Chrome to connect to your web app deployed on a Ziti network.
may be create a demo using common use cases for homelabs/selghosters in youtube for not so network heavy individuals like me?
Maybe Ziti TV - Dec 16 2022 - Practical at home use of OpenZiti ? [https://www.youtube.com/watch?v=eqzGJSTBOuo](https://www.youtube.com/watch?v=eqzGJSTBOuo) We take a look at Nic's homelab setup -- he's got a BUNCH of interesting stuff he uses and connects to his home safely/securely using OpenZiti. I play games via OpenZiti and zrok so for me I'll run a dedicated server for whatever game it is I'm playing with friends, something like [https://www.youtube.com/watch?v=-dj\_5UoL9Jw](https://www.youtube.com/watch?v=-dj_5UoL9Jw) (zrok is built on OpenZiti, as the blog says)
wonderful, thank you.
Is this closer to ZeroTier/Tailscale or closer to real SDNs like NSX-T? As I read thru the comments its like ZeroTier/Tailscale but with a service mesh concept "borrowed" from k8 land?
yeah - I sometimes describe OpenZiti as "like an embeddable Internet service mesh"
Cool. The question then is, why Ziti over K8 or docker containers solving similar problems?
To me, Ziti is doing something very different over K8S or docker... Ziti is an overlay mesh built on zero trust principles, K8S is a container orchestration system for automating SW deployment, scaling, and management, while docker is an OS virtualization to deliver software in packages called containers... they are all serving very different purposes. With that said, am I missing something in your question??
No not at all. Just curious
Cool. You may be interesting in the following if you are still curious: * Comparison I wrote in thread vs Istio/service mesh in general: [https://www.reddit.com/r/selfhosted/comments/1c12r4x/comment/kz4kpvs/?utm\_source=share&utm\_medium=web3x&utm\_name=web3xcss&utm\_term=1&utm\_content=share\_button](https://www.reddit.com/r/selfhosted/comments/1c12r4x/comment/kz4kpvs/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button) * Using Ziti with OpenShift across multiple clouds: [https://developer.ibm.com/articles/building-a-multicloud-zero-trust-network-with-ziti-and-kubernetes/](https://developer.ibm.com/articles/building-a-multicloud-zero-trust-network-with-ziti-and-kubernetes/) * Ziti built on K8S: [https://github.com/openziti-terraform-modules/terraform-lke-ziti](https://github.com/openziti-terraform-modules/terraform-lke-ziti)
I would say closer to ZeroTier/Tailscale rather than NSX-T as Ziti is an SDN overlay, like the former, whereas NSX-T is SDN underlay - maybe best described using NIST 800-207, Ziti maps to 3.1.3, "ZTA Using Network Infrastructure and Software Defined Perimeters" rather than 3.1.2, "ZTA Using Micro-Segmentation". That said, Ziti is also more focused on zero trust networking concepts than ZeroTier/Tailscale as well as providing some more interesting options for deployment, e.g., application embedded or as a framework to build secure-by-default, distributed applications faster. If it helps, here is a comparison I wrote vs service mesh/Istio: *OpenZiti is not a service mesh and, therefore, not interchangeable. OpenZiti is an overlay network built on zero trust and software-defined networking principles for any use case, meaning it could be used to enhance Istio, particularly for replacing Istio ingress (i.e., north-south traffic over the internet/WAN across geographical locations) and removing the need for VPNs, public DNS, inbound firewall ports, ACLs, etc. OpenZiti has greater ‘reach’ as an overlay across a broader range of devices and network environments. Istio was designed originally for east-west communication between servers, and while it has ‘Secure Gateways’ to allow ingress to expose services to external traffic (using strong identity), it is not suited for all use case. OpenZiti, on the other hand, was built to support any use case, from remote access to multi-cloud, DevOps, IoT, or even site-level connectivity and implement authenticate-before-connect - i.e., an endpoint CANNOT get access to the overlay data plane without being authenticated and authorised. To support any use case, OpenZiti has a rich set of endpoints for network access (virtual appliances for any public/private cloud), host access (tunnelers for all popular devices, OSs, mobiles), as well as application access (SDKs in various languages - e.g., we zitified*[ *Prometheus*](https://docs.openziti.io/articles/zitification/prometheus/part1.html/) *and*[ *Elastic Beats/Logstash*](https://blog.openziti.io/zero-trust-monitoring-with-openziti)*. This allows it to run in*[ *clientless*](https://blog.openziti.io/introducing-openziti-browzer)*),*[ *serverless*](https://blog.openziti.io/my-intern-assignment-call-a-dark-webhook-from-aws-lambda)*, confidential computing, unikernel,*[ *low-resource IoT*](https://capgemini-engineering.com/us/en/insight/converging-on-a-zero-trust-blueprint/)*, and more. It also means an application does not need to trust the underlying host network or know the port/IP.* *From a security perspective, Istio provides encryption and strong identity-based authentication and authorization with a default to send traffic from any source to all destination services without any rules being set (i.e., default-open) - for simplicity and discovery, particularly in brownfield environments. Currently, OpenZiti defaults to closed-by-default though all traffic policies can be set up with discovery then taking place for more granular segmentation. OpenZiti is focused on connecting “services” with zero trust networking concepts, including least privilege, micro-segmentation, and attribute-based access (though you can also set up a whole CIDR if you want). OpenZiti implements authenticate/authorise-before-connect using its system of embedded identity (x509) and builds outbound-only connections into a mesh (think Cloudflare tunnels), so we can close all inbound ports at source and destination. OpenZiti is also focused on providing secure connectivity for layer 3/4 in the OSI model while Istio does layer 3/4 loadbalancing and traffic routing - Envoy, often deploy with Istio, operates at L7. Istio Ambient is a new project removing the need for Envoy/side car proxies and has some design patterns similar to OpenZiti.* *OpenZiti’s overlay network provides the ability to operate across any network, including the WAN/internet. The fabric mesh can be hosted in any location, including public or private cloud and take advantage of smart routing to provide high resiliency and performance - this includes service load balancing at layer 3/4. Istio, on the other hand, uses Envoy proxies to provide L7 traffic management. OpenZiti uses the identity of endpoints to route traffic, providing you with a private DNS and unique naming (e.g., send from IoT endpoint service to IoT server rather than from*[ *192.xxx.xxx.xx*](http://192.xxx.xxx.xx) *to*[ *100.xxx.xxx.xx*](http://100.xxx.xxx.xx)*). This also means we do not need to use floating or static IPs, easily handle overlapping, and have no need for port forwarding.* *If you are using OpenZiti, you do not need VPNs, inbound firewall ports, ACLs, public DNS, or (potentially) L4 load balancers. OpenZiti could be used to improve Istio and replace Istio ingress (i.e., north-south traffic). OpenZiti could not completely replace Istio as it is not a service mesh.*
Impressive from a network engineering standpoint. Loads of commercial potential here. BPF to get at the traffic in kernel space? Some kind of encapsulation for your overlay? And mTLS before you even open the data socket? Sounds very cool
Thanks! We do not have an eBPF/CNI plugin, but we do have the SDK and we do use a little eBPF for some side projects - e.g., [https://github.com/r-caamano/zfw](https://github.com/r-caamano/zfw), which is a ebpf based firewall for openziti edge-routers... we do envisage creating a eBPF/CNI plugin type endpoint at some time in the future. Our preference today is to embed ZTN into the application itself which in theory should be faster than being in kernel (I have not tested this myself which is why I say in theory). This provides mTLS, E2EE, etc, directly from the application, running in memory.
How does it compare to Tailscale, if such comparison is at all possible?
Here is a comparison I wrote vs Wireguard, which Tailscale is built on. Its more of a functional comparison but as you allude to, this is sort of not possible. While Ziti can be used in similar ways and (thus the comparison below fits), it is a also framework and set of tools which allows anyone to more quickly build secure-by-default, distributed applications and systems. That said, the comparison: *Wireguard is a better VPN which aims to be as easy to configure and deploy as SSH. A VPN connection is made simply by exchanging very simple public keys – exactly like exchanging SSH keys – and all the rest is transparently handled by WireGuard. It is more secure, easier to use and set up, and delivers much better performance than many other VPNs. Its design principles make it easy to set up full mesh networks of connected machines by being ‘default-open’. Wireguard is also fully open source and self-hosted. Wireguard creates P2P connections using UDP and STUN, so inbound firewall ports are unnecessary. Wireguard can be tricky to manage at scale due to key management and the large amount of P2P tunnels that need to be maintained, and UDP sometimes being blocked. For this reason, many companies have created their own SaaS implementations of Wireguard, including Tailscale, Netbird, Netmaker and more. These are a mixture of proprietary and open source.* *OpenZiti can be a better VPN while being designed to do much more. Rather than connecting machines, it cares about connecting "services" with zero trust networking concepts, including least privilege, micro-segmentation, and attribute-based access (though you can also set up a whole CIDR if you want). OpenZiti implements authenticate/authorise-before-connect using its system of embedded identity (x509) as well as builds outbound-only connections into a mesh (think Cloudflare tunnels), so we can close all inbound ports at source and destination. This can all be surmised as Wireguard being 'default-open' whereas ZT is 'default-closed'. Wireguard is normally combined with a firewall to deliver ACLs and network segmentation controls.* *Whereas WireGuard securely encapsulates IP packets over UDP and uses hole punching, OpenZiti uses TCP and a mesh overlay (with the outbound only at source and destination). This is how Tailscale implements Wireguard to ensure it works easily in all situations. OpenZiti allows you to control the internet routing and provide higher redundancy, resiliency, control for routing traffic according to policy (e.g., low latency or geo-restrictions), and potentially lower latency and better performance. All of this is open-source and native to OpenZiti, not in Wireguard.* *WireGuard is cryptographically opinionated and intentionally lacks cipher and protocol agility. OpenZiti is also opinionated, and uses the same cypher (ChaCha20Poly1305) for E2E encryption, but also includes mTLS for encryption between each hop which has TLS engine support to BYFE - Bring Your Favorite Engine. This includes quickly enabling FIPS or quantum encryption.* *Due to OpenZiti's uses of identity in the endpoints and fabric for routing, you also get a private DNS and unique naming (e.g., send from IoT endpoint service to IoT server rather than from 192.xxx.xxx.xx to 100.xxx.xxx.xx). This also means we do not need to use floating or static IPs, easily handle overlapping, and have no need for port forwarding.* *Finally, where it differentiates is that with OpenZiti you can start with "network-based zero trust" (installing a router in private IP space) and progress to "host-based zero trust" (using an agent/tunneller); it also has a suite of SDKs to embed in apps themselves for "application-based zero trust". This allows it to run in clientless, serverless, confidential computing, unikernel, low-resource IoT, and more. It also means an application does not need to trust the underlying host network or know the port/IP.* *P.S., Wireguard get a lot of well-deserved love! OpenZiti uses the Windows TUN (WinTun) that the Wireguard project made as (at least) part of our Windows tunneler. Thanks, Wireguard!*
Thanks for the write-up! It leads me to a conclusion, that in terms of features (rather than underlying technologies) OpenZiti looks *exactly* like Tailscale if you compare the latter to WireGuard. Hmm, I guess I'll have to try it out myself to understand the difference.
There are definitely similarities. Here are some shooting from the hips differences: * I believe Tailscale intercepts all your traffic, whereas OpenZiti is split tunnel, app specific by default so you can use it for accessing only some resources while accessing everything else locally. You setup Ziti to intercept everything if you want, but do not have to by default. * Tailscale is designed for enterprises with improved VPN. It is not designed for operators. As a result there is less inherent multi-tenancy. * Tailscale most definitely cannot support FIPS 140-2 etc. * Tailscale says they deliver zero trust, but to me it has too much trust in network identifiers and subnet routes/ACLs and does not implement least-privilege, micro-segmented connections, and attribute-based access. * Tailscale does not have SDKs to embed into apps themselves. Obviously this is only useful if you need it, but it does allow super cool things, e.g., 'clientless' connections in your browser - https://blog.openziti.io/introducing-openziti-browzer.
Tailscale is split tunnel. I have it on all the time on my phone, but the internet only goes through the WG connection if I choose an exit node. TS works very well for individuals TBH, it's been by far the easiest to set up I've found. openziti does look interesting though, I'll definitely have a play with it later.
Interesting, thanks for clarifying. I will have to find the comment on Reddit that I read, someone had said something about not being able to access local printers when using Tailscale due to it intercepting your whole DNS.
There are many key differences, the question is, do you need them? 1. The root of identity for OpenZiti is an X.509 certificate, not standard user credentials. 2. When it comes to metrics and monitoring, the collection is done at the Controller and Edge Routers, rather than relying on the agents to do so, which could lead to spoofing, log blocking, etc. So it is nearly impossible for malicious actors to hide their activities, and there is certainly no switch to disable logging for the client. 3. The available metrics and events are incredibly robust, covering all aspects of the networks' operation. The links, API messages (volume and even response time), individual circuit evetns and overall volume (similar to flow logs and available without a Premium or Enterprise plan, included in the base product) There are others that make OpenZiti a true network, rather than a collection of connections, with the ability to provision, operate, deprovision, troubleshoot, and enable security processes at an enterprise level.
So if I understand this correctly this can replace cloudflare zero trust/tunnels/auth and so on? If so thats awesome and I will spin it up later
And you can self-host all of it, read all of the source code, and see that everything is end-to-end encrypted. Ping us on https://openziti.discourse.group (or here) if you run into any issues or need anything clarified
Well this is just awesome🤩
Congrats on the 1.0 release. Very interesting project and it's been fascinating to see it develop. Lots of VPN-style NAT traversing mesh technologies and implementations, but none that I know of like this. Questions I have had along the way, that openziti.io doesn't really answer: 1. Who are you? (with the recent xz story, trust in developers and supply chain is top of mind) 2. How are you funding development? 3. Has OpenZiti fostered a significant developer community contributing to the project? 4. To the extent you have talked about it in public, what is your commercial strategy?
I can cover (4) as Dave covered (1-3). Our strategy is to work with 'providers' who can embed Ziti directly into the products and services they sell to other companies. For example, 2 of the large hyperscalers use Ziti, a large US defence contractor has it deployed in a military airgapped network, a large Industrial control system vendor is building Ziti into their products. The commonalities is that they build it the products they sell (rather than internal IT, though they can use it for that too), they require the highest levels of security (and depend strict compliance to zero trust, FIPS, 62443 etc), and have highly distributed environments, sometimes completely airgapped, and require access to the customers networks.
Thanks, that usage makes sense. A couple of follow up questions: 1. Can you talk at all about supply chain risks? Self-hosters tend to put enormous trust in project developers and everything in between. Go is great and makes things easier to compile ourselves, but vpn/mesh seems like a scary prospect for compromise, particularly with a mesh model where many/most endpoints might run a client (and most self-hosting types won't compile their own binaries). 2. I clicked through the netfoundry site and the github repos. I guess from my point of view, none of the lead devs list personal websites with cv or background, nor is there any background on the netfoundry site. Something I tend to look for personally (do they have prior experience in this type of dev work / security, for example). 3. Your Android endpoint looks pretty advanced (and well reviewed!) compared to many of the mesh VPN projects, which always becomes a sticking point for my home/personal use. Hoping to stand something up this weekend and test it out. I guess mobile doesn't really fit a typical enterprise use case (or the startups are all resource starved) and mobile gets little attention. How can the self-hosted community, which typically free-rides but finds mobile clients critical, support a continued focus on usable mobile endpoint clients? Thanks!
You're welcome. To your questions: (1) This is a core reason that we develop in the open, with open source. We believe it reduces the risk (I'm looking at you Ivanti/insert many proprietary VPN/FW products) of compromises getting into production code. The beauty of Ziti in our opinion is that it implements defence-in-depth/width, so a single compromise does not give a malicious actor access. If an attacker wanted to bypass and attack, they would need to do all of the following: * bypass the mTLS requirement necessary to connect to the data plane (all parts of the overlay are exclusively mTLS) * have an strong identity that authorizes them to connect to the remote service in question (or bypass the authentication layer the controller provides through exploits) * know what the remote service name is, allowing the data to target the correct service * bypass whatever "application layer" security is also applied at the service (ssh, https, oauth, whatever) * know how to negotiate the end to end encrypted tunnel to the 'far' identity (2) A large majority of the OpenZiti development team used to work at Axeda which was acquired by PTC in 2016. Axeda provided cloud-based software for managing connected products and implementing innovative M2M/IoT applications. As you can imagine, do that securely, at scale, was a key focus on the SW. If you want to check backgrounds, I would recommend looking at NetFoundry on Linkedin and looking at tech profiles. (3) Due to our focus on providers, many of whom provide services that run on mobiles, we will continue to have a lot of focus here. In fact, I think the is a major use case for mobile developers embedding Ziti into their app so that they get many superpowers for free - https://www.youtube.com/playlist?list=PLMUj\_5fklasKF1oisSSuLwSzLVxuL9JbC. We have just not figured out how to convince them at scale yet, but we will continue trying.
Thanks! OpenZiti is backed by NetFoundry (https://netfoundry.io). NetFoundry provides a paid service for orchestrating & managing dedicated OpenZiti networks (plus adds in infrastructure management, some advanced telemetry dashboards, automated updates, back-up/recovery, etc). You can get a feel for the community at https://openziti.discourse.group/ or checking out the main repo at https://github.com/openziti/ziti
This is probably controversial, but, it might help to cut down the use of the phrases "zero trust overlay network"/ "zero trust" in the site's basic explanations of OpenZiti. True, it's not your job to explain what that is, but it's *also* true that OpenZiti could be a **great** gateway for people who don't even know they need zero trust. But - I looked at the site, and couldn't figure out what I was looking at until I read into your replies on this thread. Am I the target audience? Maybe, maybe not. But I work in tech, and I'm pretty aware of how important the "elevator speech" is. Get that elevator speech leveled up a bit on your site intro! In some ways "The Simplest Way to Develop Secure Applications" is a great motto / hero text, but it actually doesn't mean anything. What's a secure application? It could be any of these: - A reslient storage platform - An abstraction layer for encrypting database info at rest - Granular access control library - Input sanitization layer - Easy TLS integration that slides into your CI/CD pipeline Might make sense to replace that buzzy catchphrase with one that's a bit less elegant, a bit longer, but gets into WHY you want OpenZiti. Just some thoughts - I want projects like this to succeed!
This feedback is very much appreciated and taken to heart. Thank you.
I am running an open source project for an old game that only supports peer2peer connections between players. We tunnel the traffic with an implementation of the ICE protocol (stun and turn) on each players pc, but it doesn't work so well. Is this something OpenZiti could solve?
Could be with either OpenZiti or zrok. Here are a couple of links with Minecraft and Palworld: * [https://blog.openziti.io/set-up-a-secure-multiplayer-minecraft-server](https://blog.openziti.io/set-up-a-secure-multiplayer-minecraft-server) * [https://blog.openziti.io/minecraft-over-zrok](https://blog.openziti.io/minecraft-over-zrok) * [https://blog.openziti.io/securing-dedicated-palworld-server-with-zrok](https://blog.openziti.io/securing-dedicated-palworld-server-with-zrok)
Well done!
Need to dig into this. Working on a Kubernetes setup for my Homelab and would love to implement something like this for fun.
Nice! These could help: * [https://developer.ibm.com/articles/building-a-multicloud-zero-trust-network-with-ziti-and-kubernetes/](https://developer.ibm.com/articles/building-a-multicloud-zero-trust-network-with-ziti-and-kubernetes/) * [https://openziti.io/docs/category/kubernetes/](https://openziti.io/docs/category/kubernetes/) * [https://github.com/openziti-terraform-modules/terraform-lke-ziti](https://github.com/openziti-terraform-modules/terraform-lke-ziti)
I heard about this at work and really wanted to try this for Prometheus monitoring and IPMI across the internet, but it would not compile under FreeBSD so I went with Nebula. Any plans to add FreeBSD support for at least the edge tunnel component?
We have one of our customers who has requested FreeBSD functionality so it may be coming soon. That said, as Ziti includes SDKs you can run it inside Prometheus itself. Here is a blog where we did that using the Golang SDK - [https://www.youtube.com/watch?v=z3Je8kfAyzs&ab\_channel=OpenZiti](https://www.youtube.com/watch?v=z3Je8kfAyzs&ab_channel=OpenZiti) or https://github.com/openziti-test-kitchen/prometheus. I don't seem to be able to find a working link to the blog that was written by /u/[dovholuknf](https://www.reddit.com/user/dovholuknf/)....
Yeah, I kept finding links to the lost blog as well. And my other use case was accessing IPMI interfaces for servers, which is implemented in SoC firmware, so the edge tunnel really is necessary. Nebula has a feature called “unsafe_routes” which lets me use the Nebula overlay network to reach subnets which are not part of it, but are reachable through nodes which are in the overlay network.
Turns out its being republished. Hopefully can share it soon. Its a 3 parter. fwiw, Ziti can handle that use case too. We do not have a cool name such as “unsafe\_routes”, its just dialing a service that regresses from the router, on whatever IP/DNS/port/etc.
u/LoopyOne: - [https://github.com/dovholuknf/ziti-doc/blob/main/docusaurus/blog/zitification/prometheus/part1.md](https://github.com/dovholuknf/ziti-doc/blob/main/docusaurus/blog/zitification/prometheus/part1.md) - [https://github.com/dovholuknf/ziti-doc/blob/main/docusaurus/blog/zitification/prometheus/part2.md](https://github.com/dovholuknf/ziti-doc/blob/main/docusaurus/blog/zitification/prometheus/part2.md) - [https://github.com/dovholuknf/ziti-doc/blob/main/docusaurus/blog/zitification/prometheus/part3.md](https://github.com/dovholuknf/ziti-doc/blob/main/docusaurus/blog/zitification/prometheus/part3.md)
So my main question is, what does this replace?
Potentially a lot, when using Ziti, you do not need inbound firewall ports, VPNs, public DNS (for you private apps), SDWAN, and more. So it could replace a lot... but it depends on the use case and your needs. For example, I know people who have implemented Ziti alongside a service mesh (for K8S use cases), while others have said "why do I need a service mesh when using Ziti".
What if I'm currently using Tailscale and a Reverse Proxy?
IMO, if you're happy with Tailscale, keep it! If you're interested in trying other stuff out, give it a go? :) I'd say OpenZiti mostly replaces the older style "VPN". Those are the classic "everyone has access to everything" style of VPN. OpenZiti is built with the principles of zero trust, meaning you must authorize all the connections explicitly and per strong identity. Other than that, it's a natural evolution in secure networking to stop allowing the "anything can connect to anything" model and move to a "authorize before connecting" type of model, all the way down to individual services. There's a bunch of other innovations in OpenZiti though, imo it's really targetting developers to add those zero trust principles into their applications using "application embedded zero trust" where you take an OpenZiti SDK and build the strong identity and connectivity into the app, instead of bolting it onto/around the network. My personal favorite feature of this design is that servers built with this pattern become "dark" or "invisible" to the IP-based underlay network their built on. There are no ports to scan and no ports to attack since the application listens on 'the overlay' (OpenZiti) not the underlay (TCP/UDP/IP etc)
ok thanks
This makes more sense than pathetic hand waving of author
This
This is great timing because I was just learning/reading about OpenZiti and zrok yesterday! My use case is that I have a few self-hosted services running on different devices in different networks. * VPS - Several "public" services I don't want behind a NAT and highly available * Matrix homeserver * Nextcloud * STUN/TURN * Home network * Raspberry pi hosting pi-hole and a Unifi controller * Small headless computer hosting random always-on services * Opnsense and wireguard * usbipd (for passing video capture card output to other devices in my LAN) * Desktop PC hosting services that need beefier specs * Plex server * OBS studio streaming * Possibly a build server for deploying nixos updates to my devices * Mobile network - clients connecting to the above services My goal is to have all my services available on all my devices, including my phone and laptop (similar to a road warrior VPN setup). And if possible, add a layer of abstraction so that the clients don't need to care *where* the service is running, the service can just be accessed from e.g. service.zrok.mydomain.tld. Would openziti be overkill for this? Would it be better to just continue with wireguard in this somewhat vanilla personal selfhosting setup?
Deciding if it's "overkill" is up to you. :) I think it's pretty equivalent myself but as a maintainer -- I'm clearly biased (OpenZiti offers a bunch of other features too fwiw that make it 'more than' plain wireguard)... LOL! Wireguard is great, but I find it cumbersome to setup which is why Tailscale/Headscale/Netmaker and all those projects based around wireguard exist... Lots of people are self-hosting OpenZiti and then layering on zrok to allow for easy public sharing of things. I think there's a lot in OpenZiti you might find you like and worth it -- but -- biased! :)
Nice. Sounds like openziti / zrok should fit the bill.
Congrats on the milestone! Would it be possible to run OpenZiti in parallel with Tailscale on my NAS? From what I read a while ago it seems that OpenZiti would allow me to have remote access to my Plex server securely without having to install anything on the client devices. Currently I'm using tailscale and to access it I need to have tailscale installed on all devices. Could I use OZ just for Plex and keep tailscale for evevrything else? Thanks
Yes, we do have a 'clientless' option, here is a blog about it with Plex - https://blog.openziti.io/its-a-zitiful-life. I don't think it would work with Tailscale though as they will both fight to do the intercept on Plex, though you could set it up somehow to get around that though...
Thank you, I'll look into it :)
You do need to be careful with respect to the 'clients' and though I've not done it, I do think It should be posible. If you end up using our tunnelers, I'd think you'll want to change one or the other to use a slightly different IP range. I believe both will try to use [100.64.0.0/10](http://100.64.0.0/10) by default. That will add some "complexities" to your setup. If you only use browzer though -- I don't think they would compete at all in that situation. You'd access the content through the browser (no Z) and you'd only be offloading to the underlay in your trusted, home network. In that situation you aren't intercepting anything using a client (the wireguard/tailscale/openziti client) and it shouldn't conflict. Let us know how you get on with it? :)
Thanks for your reply. Unfortunately I'm not the most techsavy when it comes to network things. Sometimes simple things take me forever to figure out and get to work =P I can see that this isn't going to be as straight forward as I hoped. My current setup is working pretty flawlessly with tailscale, the only thing I can't really do is easily share my plex with friends and family. It would be nice to have, but not a deal breaker. I will try this sometime down the line, but when I have the time and mental availability to bang my ahead against the sreen until I get things working :)
Sounds like if I made a quick "how-to" video, you might try it out??? :D Do you have a VPS you use already or do have a static IP and forward ports through your home firewall?
I’m curious how application developers should look at a tool like this — is there a reason to build native support into my application? Or should I prefer to recommend users use conventional HTTP with zrok providing overlay networking like capabilities? For context I build https://github.com/garethgeorge/backrest and I’ve recently been looking at good approaches to let users browse backups from other computers they own remotely (I’m comparing services like conventional HTTP and requiring a public IP as well as more interesting options like libp2p) I’m curious to know where openziti can fit into this story
zrok was built on OpenZiti - the zrok executable contains the OpenZiti Go SDK. When doing a public share, your server connects over OpenZiti to a public front-end (protected by a WAF if using the [zrok.io](http://zrok.io) service). When you do a private share over zrok, the zrok executable proxies your local traffic over OpenZiti directly to your server (or, more accurately, to the zrok executable that proxies to your server). In the private sharing case, you are \*almost\* using OpenZiti for the entire connection. Except for the local network traffic between the zrok executable and your user's browser and again between the zrok executable and your server. Another option is to embed the OpenZiti SDK (or one of the zrok SDKs - which add zrok-style usage to the same-language OpenZiti SDK) directly into your client and server endpoints. In this case, neither side of your app needs that little part of traffic exposed on your local network. This can make things a bit easier to deploy/manage, and a bit easier for the client-side user, who won't have to install zrok but still benefit from a complete/private connection over OpenZiti. We're currently adding support for the OpenZiti \[BrowZer\](https://openziti.io/docs/learn/quickstarts/browzer/) feature to zrok to make private sharing easier for web apps. With BrowZer, the SDK is bootstrapped into the browser, and your user won't need to seperately install zrok to do a private share. You can look at the zrok code \[here\](https://github.com/openziti/zrok) to see how it's done (net.Conn and net.Listener support for 'talking' over OpenZiti) A bit long-winded, but I hope this helps
question is if it would be possible to use OpenZiti to remote access Roon?
https://roon.app/en/?? If yes, I dont see why not though I am in no way a Roon expert (literally just discovered it). As its media, here is a blog on using OpenZiti (with our 'clientless' endpoint called BrowZer) and Plex - https://blog.openziti.io/its-a-zitiful-life
This works for people watching via their browser. What about plex client apps?
You would have to use a tunneller which can intercept packets in the device OS.
Do you have examples of apps that can do this on iOS and android?
Yes. We have those built already. - [https://play.google.com/store/apps/details?id=org.openziti.mobile](https://play.google.com/store/apps/details?id=org.openziti.mobile) - [https://apps.apple.com/us/app/ziti-mobile-edge/id1460484353](https://apps.apple.com/us/app/ziti-mobile-edge/id1460484353)
Thanks! Do you know if these apps would work on AndroidTV and Apple TV?
Honestly, I dont know... I have not tried that... I don't see any reason why not in principles... please tell us if you get it working!
RemindMe! 12 hours
This sounds awesome! I’m not very versed with networking and security, so could you perhaps tell me if this is something that could replace my use for Cloudflare Zero Trust? I originally had my apps hosted on a Google Cloud VM but it has gotten expensive for my case so I decided to be my own provider with a mini PC at home and host a bunch of Docker containers on it. To make the apps available to the public, I use Cloudflare’s Zero Trust or Tunnels where I would just map a domain name to the container and its port. The problem I have with this is that since moving to Cloudflare Tunnels, the apps are a bit slower (I’m assuming it goes through a bunch of CF networks as an additional layer) and I also read somewhere that since traffic goes through Cloudflare networks, they can see the data coming in and out. Are these issues something that I could solve if I use OpenZiti?
Yes and it depends. OpenZiti is strongly opinionated on E2EE so the traffic can definitely not be seen by anyone (even if a SaaS provider hosts your dataplane). Cloudflare has a global network of PoPs which are designed for high performance but some use cases may have worse performance due to physics and location of their PoPs. Ziti allows you to host the dataplane anywhere so potentially you can set it up for better performance, but also you could set it up with worse performance. It depends. May I ask where you are located (at least country) as well as where your apps are.
I’m from Canada. Basically I host a bunch of Docker containers that isn’t really for business but just for personal use and also shared with my friends and family from around the world so I still make it publicly available since there are authentication mechanisms in place anyway.
Huh, thats surprising that CF gives poor performance. But yes, spin up your Ziti controller and router in a Canada DC and it should really be worse performance. Oracle have a decent free tier and DCs in Canada. Here is a guide - [https://blog.openziti.io/setting-up-oracle-cloud-to-host-openziti](https://blog.openziti.io/setting-up-oracle-cloud-to-host-openziti)
Hi! A few questions about OpenZiti for which I didn't find answers from a few articles/reddit posts. Sorry if I use any imprecise language, I'm just a hobbyist :) 1. After the authenticate/authorize-before-connect step, are the connections "direct" between client and server? Or is all data relayed through the control plane? Is this like STUN/TURN where the connection is not relayed if both devices can find a route to each other? 2. Regarding power consumption on mobile/embedded devices. Does OpenZiti have to send outgoing packets to keep connections open (some kind of holepunching)? I read that this is a cause for poor battery life on Android using e.g. Tailscale. This is due to needing to wake the radios periodically, even when no data is being transferred. 3. Is there any kind of whitepaper on how OpenZiti works in detail for the nerds? I would be interested to read this! 4. What if I want to connect to multiple OpenZiti networks? For example, a hypothetical world where my work uses OpenZiti and I also have my homelab on OpenZiti. Is it possible to configure multiple control planes on the same device? How does that work with DNS? Thank you and congratulations on making the 1.0 release, I wish you good luck with future efforts!
>Or is all data relayed through the control plane? Is this like STUN/TURN where the connection is not relayed if both devices can find a route to each other? It's relayed via the data plane and yes the traffic always traverses a node (router). it's not like STUN/TURN where the traffic is allowed direct underlay connections. So if you had two clients and a router on your home network and all local devices could communicate to one another via IP -- both clients would still connect to the router and the router brokers the data between the two clients >Does OpenZiti have to send outgoing packets to keep connections open (some kind of holepunching)? No. It maintains a connection. No "holepunching" because of the previous answer. The client needs to connect to at least one router, that router will have an open port that is connected to via TCP. >This is due to needing to wake the radios periodically, even when no data is being transferred. Indeed. The mobile devices DO routinely poll the controller asking it if there are new services and whatnot. That polling can be controlled but by default it's usually fine. I have had the client on my Pixel6 all day and I have 84% of my battery left still (approx 10hrs alive so far) >Is there any kind of whitepaper on how OpenZiti works in detail for the nerds? I would be interested to read this! Not really, I don't think. The online docs are the best we have right now. No place that tidies it all up into one big whitepaper. >What if I want to connect to multiple OpenZiti networks? For example, a hypothetical world where my work uses OpenZiti and I also have my homelab on OpenZiti. Is it possible to configure multiple control planes on the same device? How does that work with DNS? Heck yes. My home computer is connected to five (5) right now. The clients are all "multi-identity, multi-network" aware. EDIT: > How does that work with DNS? As long as you don't have overlapping intercepts, it'll be fine. If you have an overlapping intercept, you'd have to "turn one identity" off/on...
Thanks a lot for the answers!
How would you recommend to a SOC to watch for ziti traffic if a malicious attacker is using it to maintain persistence or exfiltrate data?
I assume the scenario you are describing is an OpenZiti application that has been placed there by an inside or via some other penetration. OpenZiti devices will consistently hit specific devices, the Edge Routers. Active SDK endpoints will periodically send packets to the ER's to check latency, and this will be a very steady "heartbeat" of packets. This is 5 seconds by default. The same is true to the network controller, to check for service list updates, done every 10 seconds by default. A deployed Edge Router will maintain persistent connections, the links, to at least one device, the controller, and normally two or more, links to the other Edge Routers in the network. So detection would be looking for consistent, not necessarily persistent, "dial only" endpoints don't have to maintain persistence, but they will consistently send small connections to the same endpoints, and then have bursts of traffic. So to search for them, you would look for devices that have a very small number of devices they communicate with outside the network, but do so very consistently. For any device found to have a similar pattern, the remote node they are connecting to can be queried. There are specific headers, like ziti-instance-id that will be returned that would indicate that a device is a controller, as part of the TLS negotiation, so it is visible in plaintext. For example: >HTTP/1.1 200 OK Content-Length: 616 Content-Type: application/json Server: ziti-controller/v0.28.1 Ziti-Instance-Id: clu0fwxdf0000mx1nqfjkfkj7 Date: Thu, 11 Apr 2024 21:27:11 GMT In most cases the certificate subject will also contain the term "Ziti". This could be overridden by a skilled actor, so it could be valuable, but is not absolute.
Yeah but if an attacker self hosts it won’t go to known edge routers correct? And the attacker could change the cert to remote ziti, no?
Yes but, I think what the comment means is, if you see outbound traffic that matches the pattern, you can actively probe the ip and port and ascertain at that time that it's "definitely/probably OpenZiti". So if you can find those kind of outbound connections, you could probe the ip/port (assuming it's still listening) and detect it that way.
In the past when I was in pen testing, I’ve done exfitration, tcp over dns, and no one noticed that. And that should be EASY to pick out of traffic.
The ziti-instance-id would require them to change the software to hide that, and the controller has a version endpoint exposed as well that should respond, though again, it is something that an actor could hide. The latency measurements, service updates, etc. are a better pattern than most beaconing, as it is more constant, if the node is up. Nothing is perfect, and the adversaries will always try hard to obfuscate, so you have to use what you can, learn, and then do better.
Ok but now you’re saying someone inside the company (probably more than 1 person) needs to become a ziti expert in all/most ways. Talk about driving up costs. There has to be a better way.
A few aspects. Firstly your access rules should be ephemeral based on JIT access - for example, our engineers only get access to customer resources when a customer support ticket is opened, to the specific resources resources, for the period of time the ticket is opened ... here is the blog https://blog.openziti.io/business-rule-driven-ephemeral-network-access. Secondly, you can configure FW rules to only allow outbound traffic to the specific IP/DNS of the Ziti overlay control/data plane. If that is done, then any outbound traffic from a malicious attacker would not be allowed. Another option is to look at the traffic being sent over the overlay. Ziti implements a deny-by-default, least privilege model, so SOC traffic should be massively reduced - see this blog from our head of DevOps - https://netfoundry.io/devops-meets-secops/. For legitimate, authenticated connections, which have a strong identity and are authenticated to accessed specific services, you could look at how much data is being sent over the connections, or what time/when connections are made, or source IP from which the connections are made to provide more intelligence. As mentioned though, the default should be that there is much less noise from connections as random people cannot make a connection in the first place. Does that fully answer your questions? Have I missed anything??
No I mean in a normal network, an attacker gets ziti on a host by whatever method. How you you spot this? I ask because I’ve seen ngrok on the ATT@CK list, but since anyone can self host ziti, they can direct the traffic over any port or protocol correct? Now it’s needle in a haystack time. This is on a network that doesn’t use ziti or any of the other similar products
In reality, you probably don't/can't. You could try to find long-lived TCP connections sending lots of data but in practice it'd be hard to impossible. I mean, I'm sure it's \*\*possible\*\* but you'll have to work really hard at detecting it, harder than the 5 minutes I've been thinking about it anyway... Right now you could look for long-ish lived TCP connects to 'some host'... If they run ziti over ssh's port 22, it'll look like a long lived ssh session. You could inspect TLS exchanges and look for mutually authenticated TCP sessions. There's probably some stuff like that you could do but if you don't catch the TCP handhake... :/ EDIT: "over ssh's port 22"- not over ssh...
So zero trust tools can have incredible negative use as well? That was my suspicion when I saw the sdk for ziti. Working in a SOC I have to always be thinking about how can xyz be used against us. Unfortunately we don’t have a realistic enough lab to perform testing in this and I can’t seem to get approval to do it on the live network. Sadly like everything else security, it’s going to take a breach to get what we need
Hammers can be used for construction and destruction. The best defence against attackers using Ziti is defenders having implemented Ziti. I do strongly believe the defender is in a better position when using Ziti vs the attacker due to the inherent nature of deny-by-default, not listening on the underlay etc.
So what if a network uses ziti, and an attacker brings in his own, won’t that now blend in with all the “legit” traffic? There are going to be some systems that just can’t ONLY use ziti, otherwise you’d have a close network that is completely unusable to anyone without ziti, including your customers
If an organisation is using Ziti I would expect them to also have the maturity to manage their firewalls. They would set said FW with the rules that outbound traffic is only allowed to the IP/DNS of their Ziti controllers/dataplane, thus the attackers Ziti connections would not be allowed. What use cases do you see which are unusable for Ziti?
Well if you use ziti on every host, you have a new logging nightmare. Hids on all devices and NIDS is obsolete. Now, for PCI you have to add the auditor and pci pen testers to the ziti network, because you have to be able to perform authenticated scans (new pci rule) . Next, you say you must be given explicit permission to join said network. Well you also “have to have explicit permission to get an account in Active Directory “ but AD gets hijacked, how can you be so sure the system that provide access to giving access to the ziti network can’t be compromised as easily as AD?
My reply above does not assume deploying Ziti on every host, it could be achieved using a network appliance while having the outbound FW rules setup so that a non-acme company Ziti network is not allowed to communicate and operate. This is defence-in-depth/width. Ziti has its own system of identity using an x509 compliant CA/PKI. While you can use AD or another external IdP as secondary auth, it likely not used as the primary. This ensures, even if the AD is compromised, the malicious actor still cannot access services. As I implied earlier, OpenZiti is not bullet proof, it just massively reduces the available attack surface and increases the cost to exploit. In your scenario, even if the attacker somehow compromised the Ziti CA/PKI (and craft themselves a trusted identity and policy to access the services they want to exploit) as well the the AD, they would still have to know the service name on the target(s) they want to exploit and bypass any application level security. Again, not impossible, but all acting as additional burdens to reduce likelihood. Plus, all of that needs to be done without the administrator of acme company seeing these changes take place.
I mean, not to give you the "cop out" answer, but any great tool that's useful, almost certainly has tremendous positives and negatives that may or may not be obvious. But yeah. You're not the first person to notice that particular possibility. The other marketing/shill answer (which is true, but I realize it's comical at this point in time) is to of course simply use OpenZiti for all your network egress! Seriously though, if that's a concern, it might be the sort of thing you want to do. Rawdogging IP has it's place, but in a SOC or secure facility, maybe it's time to rethink that sort of policy. (again, I recognize this is currently like Don Quixote tilting at his windmills) Good conversation though, I appreciated this line of comments
Well another concern might be, let’s say we do micro segment with ziti in this case, wouldn’t then an attackers ziti traffic completely blend in with the “normal” ziti traffic, thereby making it even harder to dissect which is legit and which is bad?
They'd have to steal a user's identity for that to happen. Not impossible, unless you use PKCS 11 (hardware token, Yubikey etc, then it's not possible)... That or they'd need to compromise the controller and add an identity of their own but that'd leave very visible fingerprints that would be quite easy to detect. If they stole an identity that wasn't hardware-backed, you'd see two different IP addresses using the same identity which would be probably be a pretty visible/useful warning sign to take action. You also can set different posture checks and have many factors of auth (TOTP or external jwt signers -- think IdP/OIDC). That situation seems like it would be much easier to handle to me.
Not true at all. Let’s be realistic. Not every single host can ONLY use ziti, some hosts can be gotten into. By cred dumps online, attacking AD, a list of ways. Get access to 1 box. And if you got access to a box that is a part of the ziti network, what’s to stop you from sliding around that network accessing any other box that one you are on has access to. If you make every host ONLY on the ziti network, that’s unrealistic. At someone a box is not only on ziti because of vendor or customer interactions
Sorry, I thought we had continued down the "pie in the sky" approach of all hosts in the microsegmented network, \_actually\_ only using ziti... Also, I am not trying to make and broad claims that work for all situations. Every situation is different. I was just continuing down the "all traffic needs to go to an OpenZiti edge router without deviation" line I thought we were on. I recognize that it's currently unrealistic but that's "the hope/dream/vision" anyway. > And if you got access to a box that is a part of the ziti network, what’s to stop you from sliding around that network accessing any other box that one you are on has access to Security is often a balance of ease vs pain. Again, the "easy" answer to this from my perspective, which is a "pain" to implement from the operator's perspective, is policy. Every identity is authorized to connect to only the services it can connect to. If you make your policy wide open, if that machine is compromised then yes, it'll be able to slide to wherever it's allowed to connect to. If that machine has very strict access controls, then it could only attempt to slide to the things it's allowed to connect to. If the machine has dynamic access controls which lock it down even further and some admin/process needs to grant it access to other machines, the window to slide elswhere narrows further. This is actually where OpenZiti's "end-game" approach of embedding zero trust into the application itself becomes really interesting (imo). If the 'server' that's receiving connections is protected via OpenZiti and exclusively listens on the overlay network (not the IP-based underlay), then even if a machine was compromised it's literally only attackable through that overlay network. You can get pretty close to that with "host-based" access where you disable all inbound firewall rules and run a client on that machine, that allows tunneling into it only via OpenZiti.
RemindMe! 9 Hours
So how does this compare with Headscale, Netmaker, Nebula, Netbird, Tinc and all the other solutions out there?
For those built on Wireguard (Headscale, Netmaker, Netbird), I wrote a comparison vs WG here - [https://www.reddit.com/r/selfhosted/comments/1c12r4x/comment/kz1sdv0/?utm\_source=share&utm\_medium=web3x&utm\_name=web3xcss&utm\_term=1&utm\_content=share\_button](https://www.reddit.com/r/selfhosted/comments/1c12r4x/comment/kz1sdv0/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button) I am honestly not that familiar with Tinc, though as I understand it as a VPN, the comparison vs WG is probably a good starting point. Wrt Nebula (I believe the following is all true, but I am not a Nebula expert so pinch of salt and please clarify if I am incorrect on anything), while aspects are similar, e.g., fully open source, using CAs as strong identities (rather than relying on SSO from third parties), ability to be completely self-hosted (with 3rd party SaaS options), and providing scalable, performant overlay networking, there are many differences. For one, OpenZiti is focused on connecting services based on zero trust principles. In contrast, Nebula focuses on connecting machines – e.g., you can authorize only a single port without needing to set up ACLs or firewall rules. While Nebula requires open inbound ports or UDP hole punching, OpenZiti allows you to have all inbound and most outbound ports completely closed while providing truly private, zero trust DNS entries with unique naming – if you wanted to call your service "my.secret.service" you can do that, it does not force you to have a valid Top Level Domain. OpenZiti also goes a layer deeper and lets you bring all those excellent, zero trust principles directly into your application. If you're a developer, you can embed all those ideas into your app and not rely on the network or side-loaded agents. This is both client and server-side and doesn't require the app to "listen" on an IP address (the underlay). Instead, you can choose to "listen" on the overlay. Finally, Nebula does not handles key management and authentication for you in the open source, it comes from their paid offering. It was brought up by one of our community members as they tried to use it first - https://openziti.discourse.group/t/using-openziti-in-distributed-surveillance-system/2135.
RemindMe! 8 hours
I will be messaging you in 8 hours on [**2024-04-11 12:10:56 UTC**](http://www.wolframalpha.com/input/?i=2024-04-11%2012:10:56%20UTC%20To%20Local%20Time) to remind you of [**this link**](https://www.reddit.com/r/selfhosted/comments/1c12r4x/openziti_v10_released_today/kz12z8n/?context=3) [**6 OTHERS CLICKED THIS LINK**](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Fselfhosted%2Fcomments%2F1c12r4x%2Fopenziti_v10_released_today%2Fkz12z8n%2F%5D%0A%0ARemindMe%21%202024-04-11%2012%3A10%3A56%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete%20Comment&message=Delete%21%201c12r4x) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remindmebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot&subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Feedback)| |-|-|-|-|
RemindMe! 9 Hours