SCADA network extension to remote sites

[mburtis]

So I'm pretty new around here but I come looking for advice or suggestions to research. Im the plant electrician/SCADA guy for a warer department. We have two water treatment facilities connected by a T1 line. Long story short the T1 line goes down all the time and service is terrible.

We have been looking for alternatives and the only real viable solution is to utilize the third party internet service with a VPN tunnel and firewalls and such. IT already extended the general network to this facility and they have a VLan set up for SCADA. However here is the sticky bit, our current SCADA system is all set up on the same network between the two plants, 192.168.254.xxx. IT originally thought they could do some NATing and make this all work. Turns out they couldn't and are now telling me I need to change the IPs of all the equipment at the second location to a different subnet. In theory this isn't that difficult but it's a very buggy FactoryTalk system and initial tests proved to be unpredictable. In this case there is an independent SCADA computer running at both locations which talks to all the PLCs at both locations, sorta a redundant setup.

I guess my question is, if you had to connect PLCs and SCADA computers at two locations across a third party internet connection, how would you do it? What products would you use ? How would you structure the networks? I know this is pretty far off in the networking weeds but I'm not afraid to learn a little IT.

Thanks for any input.

 

[nicherbert]

I think there are a lot of details on best approach here that are still needed to find an optimal solution, but I can clue you into a few things that I see potentially problematic that they are facing. Historically, when you used a T1 line, it uses an MPLS setup that essentially extends your ethernet network like it was just a switch between the two sites, facilitated by the ISP (essentially a VPN done by the ISP). Now, software allows us to do this with regular internet connections, and doing so with a VPN tunnel is ideal. In fact, this allows multiple VPN tunnels to be established and routed appropriately. Best practice today is for the business network to have a Point-2-Point VPN tunnel so that regular business users have a connection. Then, you typically have another firewall that protects the OT space and these devices also have VPN tunnel to/from each other that rides along the established business network. To make all this work, however, the traffic must be routable, meaning it cannot send traffic to the same network (i.e. 192.168.254.x) to 2 different locations, and hence the ask to re-ip the network.

In the ideal world, you need to have 2 different networks so you can route the traffic effectively and easily. So, a simple question about your network. You said you have 2 redundant SCADA systems that are on different sites, would the IP addresses for the devices be in different ranges perhaps? Meaning, is one site using 1-128, and the other site using 129-255 or something like that? If so, you can just make the subnet smaller (instead of using 255.255.255.0, you could use something like 255.255.255.128 to break the subnet in 2, but this only works if you happened to segment out the network somewhat. Then you could just update the subnet mask rather than all the IP addresses. You can go as small as you'd like, even to a .254 subnet which only allows 2 ip addresses in a point-to-point link.

The ideal setup would be 2 different networks that can then be routed. You can make them smaller if needed, but if you are using 10.x.x.x or 192.168.x.x networks, just make them all different and use the routers/firewalls to route traffic between them as appropriate. I'm not sure what SCADA system you are using, but if they must exist on the same network for redundancy purposes, this might effectively limit this ability (i.e., they use DCOM or some old technology to communicate with one another). For cyber security purposes, this is actually ideal too. It's called network segmentation, and limits to spreading of viruses and malware by forcing rules at each firewall along the way. If you get into SIS systems that are network connected, you might add another layer of protection with yet another firewall and another new network. This is the new norm and your systems should be designed to handle this going forward.

I am starting to see more PLC vendors use I/O networks with a 192.168.x.x network, with the main PLC IP using whatever the subnet is for the site. For larger installations, this might work, but connecting to the ethernet devices on this network becomes an issue as it's not routable anymore as every site has the same network now, and the whole point with ethernet is to make connectivity easier. In your situation, NAT'ing is typically the answer, but still requires some reconfiguration as the device would always be translating 1 ip for another to get to the other site, and communicating with the other site would use the other sites IP address (differnt network) in the program. If you had the situation with I/O networks, you could do this pretty easily, as all IP's map one for one, but in your situation, you only translate specific IP's (meaning, the firewall/router must act like it's all those IP addresses and respond accordingly) to the other, and this is why it doesn't work well, and is a maintenance nightmare that you really don't want (you become so dependent on IT to make firewall/router changes now). With those newer I/O networks, you are better off getting a new subnet each time and effectively configuring it to be unique so you can route it easily. Again, you can do this by subnetting the assigned network better (i.e. you are given 192.168.100.0 [255.255.255.0], but you could take that and make 192.168.100.0 [255.255.255.128] and 192.168.100.128 [255.255.255.128] 2 different networks that route to the same site and keep things separate on your side).

One thing I didn't ask is if these two sites were in proximity to one another. If so, you might be able to get away with a wireless bridge, which effectively extends the ethernet network at layer 2 (switching) and then you don't need routing at all. I'm guessing that's not the case here.

 

[Old Man]

I use a "Private Verizon Network", we have locations in 3 states and if any hardware needs to be connected to our Scada(Ignition) it wil use the "Private Network". There are some limitations such as this network can only handle 327675 (ip addresses), but this can be expanded by Verizon should you need more.

 

[mburtis]

These plants are about 14 miles apart with topography that makes a point to point radio or other line of sight solution not viable.

Currently it is all one big network, no duplicated IPs or anything they are just all .254.x addresses.

Sounds like in the long run splitting the network is the best solution. Essentially we are building what you describe. There is already a VPN tunnel setup between the sites for the office network. However IT has a Vlan setup for the SCADA instead of a separate physical firewall.

The SCADA system is Factorytalk and there isn't that much that needs changed. I actually had 99 percent of the equipment switched from .254.x to 225.x addresses the other day. The only problem was when I changed the IP of the virtual machine hosting the SCADA project the client machine wouldn't reconnect to the factory talk directory. I tried a few things and couldn't get it to fix itself, but I'm far from an expert.

 

[mburtis]

What is a SIS system. I'm not familiar with that.

 

[mburtis]

So are there some kinds of somewhat standardized IP schemes? It seems like 192.168.x.x is a very common address scheme. Seems I've seen mention of a 10.o.x.x scheme or something like that, that is also very common. Possibly a 174.43.x x or similar scheme.

Is there a reason these schemes are more common?

I'm thinking of actually laying out a network design and slowly moving everything towards it.
Something like 192.168.x.x where the third set of numbers is unique per location. Then set up standard ranges in the 4th set of numbers. Something like .10 through .30 is reserved for plcs, .40 through .60 is reserved for hmis, .70 through .90 is reserved for vfds, etc etc etc. Would mean rebuilding basically the entire system, but it seems like it would set it up very well for the future.

 

[Old Man]

You better make sure you understand "Networking".

Example: 192.168.100.10 will not communicate with 192.168.200.15 unless you have your subnet mask programmed correctly.

 

[mburtis]

In my example above there would be a firewall/router at each location necessitating the need for the different networks. I would have to think about how to handle the point to point radio networks which would not need the firewall/router setup.

 

[BeepBob]

At my old work place we have the same config for distance pump stations. We use a GE ORBIT MDS with cellular. The setup on the Orbit allows you to create an IPsec Tunnel to move data between two endpoints. But then it allows you to bond the IPsec Tunnel to the local interface. This way the remote network can be setup as the same subnet as your local. The nodes on the network isn't even aware of the IPsec tunnel interface.


The GE engineer can walk you through the setup. Pretty easy. End result: Everything on same subnet like you want.


The Orbit can do License or unlicense radio too