Kubernetes || Remove Namespace stuck in terminating state after delete

Currently, we are getting some issues while deleting the namespace in Kubernetes(AKS). When we run "kubectl delete ns" command, it got stuck and after sometime, if we abort this it will remain in terminating state forever.

Below is the kubernetes version we are using-

$ kubectl  version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.0", GitCommit:"ddf47ac13c1a9483ea035a79cd7c10005ff21a6d", GitTreeState:"clean", BuildDate:"2018-12-03T21:04:45Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

Here, we will see how we can use the workaround to remove the stuck namespace.
1. Create test Namespace-

$ kubectl create ns test
namespace/test created
$ kubectl get ns test
NAME   STATUS   AGE
test   Active   45s

2, Try to delete namespace. You will notice that below command got stuck there. I will hit "ctrl+c" to abort and then check status.

$ kubectl delete ns test
namespace "test" deleted
^C
$ kubectl get ns test
NAME   STATUS        AGE
test   Terminating   3m

You saw that namespace is still there in terminating state. Now in below steps we will discuss about the workaround.

1. Get the json output for above namespace-

$ kubectl get ns test -ojson > test.json
$ cat test.json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2019-01-25T11:34:16Z",
        "deletionTimestamp": "2019-01-25T11:36:27Z",
        "name": "test",
        "resourceVersion": "6575638",
        "selfLink": "/api/v1/namespaces/test",
        "uid": "25e3d87e-2095-11e9-bbb7-36dd4f9e3a95"
    },
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

2. You saw the "finalizers" object in above json file. Please remove the line with "kubernetes". So, the file will look like below -

$ sed -i  "/"kubernetes"/d" test.json && cat test.json
{
    "apiVersion": "v1",
    "kind": "Namespace",
    "metadata": {
        "creationTimestamp": "2019-01-25T11:34:16Z",
        "deletionTimestamp": "2019-01-25T11:36:27Z",
        "name": "test",
        "resourceVersion": "6575638",
        "selfLink": "/api/v1/namespaces/test",
        "uid": "25e3d87e-2095-11e9-bbb7-36dd4f9e3a95"
    },
    "spec": {
        "finalizers": [
        ]
    },
    "status": {
        "phase": "Terminating"
    }
}

3. Start the kube proxy using comand "kubectl proxy"

$ kubectl proxy
Starting to serve on 127.0.0.1:8001

4. Now on other terminal run below command-

$ curl -H "Content-Type: application/json" -X PUT --data-binary @test.json http://127.0.0.1:8001/api/v1/namespaces/test/finalize
{
  "kind": "Namespace",
  "apiVersion": "v1",
  "metadata": {
    "name": "test",
    "selfLink": "/api/v1/namespaces/test/finalize",
    "uid": "25e3d87e-2095-11e9-bbb7-36dd4f9e3a95",
    "resourceVersion": "6577114",
    "creationTimestamp": "2019-01-25T11:34:16Z",
    "deletionTimestamp": "2019-01-25T11:36:27Z"
  },
  "spec": {

  },
  "status": {
    "phase": "Terminating"
  }

5. Check is namespace has been deleted or not-

$ kubectl get ns test
Error from server (NotFound): namespaces "test" not found

Here you go!! Its deleted. This is some kind of bug in K8S. I haven't checked if this have been already fixed on higher version.