[HBase] ClockOutOfSyncException 해결
로그
2019-12-17 10:39:39,240 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Master rejected startup because clock is out of sync
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server edc1,60020,1576546764925 has been rejected; Reported time is too far out of sync with master. Time differe
nce of 32111ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2297)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server edc1,60020,1576546764925 has been rejecte
d; Reported time is too far out of sync with master. Time difference of 32111ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1269)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2295)
... 2 more
2019-12-17 10:39:39,290 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server edc1,60020,1576546764925: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server edc1,60020,1576546764925 has been rejected; Reported time is too far out of sync with master. Time difference of 32111ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
org.apache.hadoop.hbase.ClockOutOfSyncException: org.apache.hadoop.hbase.ClockOutOfSyncException: Server edc1,60020,1576546764925 has been rejected; Reported time is too far out of sync with master. Time difference of 32111ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:327)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2297)
at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.ClockOutOfSyncException): org.apache.hadoop.hbase.ClockOutOfSyncException: Server edc1,60020,1576546764925 has been rejected; Reported time is too far out of sync with master. Time difference of 32111ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1269)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
at org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2295)
... 2 more
2019-12-17 10:39:39,292 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.regionserver.LocalIndexMerger]
2019-12-17 10:39:39,344 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Unhandled: org.apache.hadoop.hbase.ClockOutOfSyncException: Server edc1,60020,1576546764925 has been rejected; Reported time is too far out of sync with master. Time difference of 32111ms > max allowed of 30000ms
at org.apache.hadoop.hbase.master.ServerManager.checkClockSkew(ServerManager.java:401)
at org.apache.hadoop.hbase.master.ServerManager.regionServerStartup(ServerManager.java:267)
at org.apache.hadoop.hbase.master.MasterRpcServices.regionServerStartup(MasterRpcServices.java:367)
at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:8615)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2170)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:109)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:185)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:165)
2019-12-17 10:39:39,344 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2019-12-17 10:39:39,382 INFO org.mortbay.log: Stopped SelectChannelConnector@0.0.0.0:60030
2019-12-17 10:39:39,389 INFO org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2019-12-17 10:39:39,389 INFO org.apache.hadoop.hbase.procedure.flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2019-12-17 10:39:39,389 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server edc1,60020,1576546764925
2019-12-17 10:39:39,389 INFO org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation: Closing zookeeper sessionid=0x16f1181ec28000c
2019-12-17 10:39:39,395 INFO org.apache.zookeeper.ZooKeeper: Session: 0x16f1181ec28000c closed
2019-12-17 10:39:39,397 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2019-12-17 10:39:39,405 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server edc1,60020,1576546764925; all regions closed.
2019-12-17 10:39:39,418 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver/EDC1/15.1.3.151:60020 closing leases
2019-12-17 10:39:39,418 INFO org.apache.hadoop.hbase.regionserver.Leases: regionserver/EDC1/15.1.3.151:60020 closed leases
2019-12-17 10:39:39,418 INFO org.apache.hadoop.hbase.ChoreService: Chore service for: edc1,60020,1576546764925 had [] on shutdown
2019-12-17 10:39:39,420 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Split Thread to finish...
2019-12-17 10:39:39,420 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Merge Thread to finish...
2019-12-17 10:39:39,420 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Large Compaction Thread to finish...
2019-12-17 10:39:39,420 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: Waiting for Small Compaction Thread to finish...
2019-12-17 10:39:39,425 INFO org.apache.hadoop.hbase.ipc.RpcServer: Stopping server on 60020
2019-12-17 10:39:39,426 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.listener,port=60020: stopping
2019-12-17 10:39:39,428 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopped
2019-12-17 10:39:39,428 INFO org.apache.hadoop.hbase.ipc.RpcServer: RpcServer.responder: stopping
2019-12-17 10:39:39,443 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2019-12-17 10:39:39,444 INFO org.apache.zookeeper.ZooKeeper: Session: 0x16f1181ec28000b closed
2019-12-17 10:39:39,444 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: stopping server edc1,60020,1576546764925; zookeeper connection closed.
2019-12-17 10:39:39,444 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: regionserver/EDC1/15.1.3.151:60020 exiting
2019-12-17 10:39:39,444 ERROR org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:127)
at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2676)
2019-12-17 10:39:39,451 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@614f43c9
2019-12-17 10:39:39,452 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook thread.
2019-12-17 10:39:39,491 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.
원인
노드간의 시간이 맞지 않아서 생기는 문제이다.
※ 시간동기화가 필요한 이유
(각종 NoSQL과 같은 형식들을 Time Stamp와 함께 보관하여 리비전 관리를 해주는 경우도 있으니, 이러한 경우 시간동기화가 필요하다.)
해결방법
NTP 서버를 설치하고 시간을 동기화시켜주면 된다.
참고
댓글
댓글 쓰기