Java – Connection denied when attempting to access a kubernetes pod through kubernetes DNS

I’m trying to set up a Hadoop single node on Kubernetes.
Oddly enough, when I log in to the pod via kubectl exec -it <pod> /bin/bash I happily access the name node on e.g. port 9000.

root@hadoop-5dcf94b54d-7fgfq:/hadoop/hadoop-2.8.5# telnet localhost 9000
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

I can also bin/hdfs dfs -put files etc so the cluster seems to work fine.
I can also access the user interface via kubectl port-forward <podname> 50070:50070 I see a data node running. So the cluster (set to “pseudo-distributed”, as described in here. Seems to work fine.

However, when I want to access my service through kubernetes DNS, I get a Connection refused

telnet hadoop.aca534.svc.cluster.local  9000
Trying 10.32.89.21...
telnet: Unable to connect to remote host: Connection refused

What is the difference between accessing a port via K8S-DNS?

The port has to be open and I can also see that the hadoop name node is listening on 9000.

lsof -i :9000
COMMAND  PID USER   FD   TYPE    DEVICE SIZE/OFF NODE NAME
java    2518 root  227u  IPv4 144574393      0t0  TCP localhost:9000 (LISTEN)
java    2518 root  237u  IPv4 144586825      0t0  TCP localhost:9000->localhost:58480 (ESTABLISHED)
java    2660 root  384u  IPv4 144584032      0t0  TCP localhost:58480->localhost:9000 (ESTABLISHED)

The full reference is my Kubernetes yml service and deployment spec.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    service: hadoop
  name: hadoop
spec:
  selector:
    matchLabels:
      service: hadoop
  replicas: 1
  template:
    metadata:
      labels:
        service: hadoop
        run: hadoop
        track: stable
    spec:
      containers:
      - name: hadoop
        image: falcowinkler/hadoop:2.8.5
        imagePullPolicy: Never
        ports:
          # HDFS Ports
          - containerPort: 50010
          - containerPort: 50020
          - containerPort: 50070
          - containerPort: 50075
          - containerPort: 50090
          - containerPort: 8020
          - containerPort: 9000
          # Map Reduce Ports
          - containerPort: 19888
          # YARN Ports
          - containerPort: 8030
          - containerPort: 8031
          - containerPort: 8032
          - containerPort: 8033
          - containerPort: 8040
          - containerPort: 8042
          - containerPort: 8088
          - containerPort: 22
          # Other Ports
          - containerPort: 49707
          - containerPort: 2122
---
apiVersion: v1
kind: Service
metadata:
  labels:
    service: hadoop
  name: hadoop
spec:
  ports:
  - name: hadoop
    port: 9000
  - name: ssh
    port: 22
  - name: hadoop-ui
    port: 50070
  selector:
    service: hadoop
  type: ClusterIP

What is the difference when accessing a port via k8s-dns?

When you call a pod

IP address, you connect directly to the pod, not the service.

When you call the DNS name of a service, it resolves to a service IP address that uses a selector as a filter to forward your request to the actual pod to find the destination, so these are two different ways to access the pod.

Also, you call the service IP address directly instead of using DNS, which works the same way. Also, the service IP address, unlike the pod IP, is static, so you can use it whenever you want.

For intra-cluster communication, you are using ClusterIP Service mode, which is the default setting, you can set it, so everything works fine here.

The current endpoint for your service forwarding request, which you can get by kubectl get service $servicename -o wide in the Endpoint column.

Regarding your current connection issue, I can recommend you :

• Check the endpoint of your service (which should have the IP address of one or more pods).

• Set the targetPort parameter for each service port, for example:

  apiVersion: v1
  kind: Service
  metadata:
    labels:
      service: hadoop
    name: hadoop
  spec:
    ports:
    - name: hadoop
      port: 9000
      targetPort: 9000 # here is
    - name: ssh
      port: 22
      targetPort: 22 # here is
    - name: hadoop-ui
      port: 50070
      targetPort: 50070 # here is
    selector:
      service: hadoop
    type: ClusterIP
  

P.S. Here is a good topic to explain How the service works. Also check out the official documentation.